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I. Real Party in Interest 

The real party in interest is Hewlett-Packard Development Company, L.P., a Texas 
Limited Partnership having its principal place of business in Houston, Texas. 

II. Related Appeals and Interferences 

Appellant is not aware of any related appeals or interferences that will directly affect or 
be directly affected by or have a bearing on the Board's decision in the pending appeal. 

III. Status of Claims 
Claims 1-51 are pending. 

Appellant appeals all rejections of the claims 1-51. 
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IV. Status of Amendments 

The Amendment that was filed on May 3 1 , 2006, has been entered and acted upon by the 
Examiner. 

No amendments were filed after the final Office action dated Aug. 23, 2006. 

V. Summary of Claimed Subject Matter 

A. Introduction 

The specification explains that the claimed invention enables automatic organization of a 
collection of objects into meaningful clusters (or groups) that may be used for any of a variety of 
applications, including permanent organization, collective annotation, and dynamic visualization 
(see page 5, lines 22-25). The objects are segmented based on criteria structured to model the 
ways in which objects typically are created so that each of the resulting clusters more accurately 
contains all of the objects relating to a common context (e.g., the same event, location, or subject 
matter) (see page 5, lines 25-29). In this way, embodiments in accordance with aspects of the 
invention can provide an efficient and accurate process for automatically segmenting a collection 
of objects (see page 5, lines 29-31). In addition, some embodiments in accordance with aspects 
of the invention provide a user with an intuitive and natural user interface for browsing and 
customizing the automatically segmented object clusters (see page 5, line 31 - page 6, line 1). 
Some embodiments in accordance with aspects of the invention also provide an automatic 
process for storing and assigning meaningful, context-sensitive names to the object clusters and 
the objects within the clusters (see page 6, lines 1-3). 

B. Independent claim 1 

The aspect of the invention defined in independent claim 1 is a method of organizing a 
collection of objects arranged in a sequence ordered in accordance with a selected dimension of 
context-related metadata respectively associated with the objects. The objects in the sequence 
are classified to generate a series of object clusters. In this process, each of the objects is 
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sequentially processed as a respective candidate for segmentation into a respective current one of 
the object clusters in the series. In addition, for each of the candidate objects, a candidate object 
interval separating the candidate object from an adjacent object in the sequence already 
segmented into the current object cluster is determined. The candidate object interval is 
measured in the selected dimension of the context-related metadata. For each of the candidate 
objects, the candidate object interval is compared to a weighted measure of cluster extent for the 
current object cluster. The measure of cluster extent corresponds to a current distance spanned 
by all the objects in the current object cluster measured in the selected dimension of the context- 
related metadata. For each of the candidate objects, the candidate object interval is compared to 
a weighted measure of object density for the current object cluster. The measure of object 
density corresponds to a measure of distribution of distances separating adjacent ones of the 
objects in the current object cluster measured in the selected dimension of the context-related 
metadata. 

FIG. 3 shows an embodiment of a method of organizing a collection of objects in 
accordance with the aspect of the invention defined in claim 1 . The objects are arranged into a 
sequence that is ordered in accordance with context-related meta data associated with the objects 
(see page 8, lines 16-17; FIG. 3). The context-related meta data may be, for example, meta data 
relating to the creation, termination, extent, operating environment, and other direct or 
circumstantial information), including one or both of time meta data and location (geographic or 
logical) meta data (see page 6, lines 9-12). The objects in the sequence are classified to generate 
a series of object clusters (see, e.g., page 5, lines 22-25, page 7, lines 22-24, and page 13, lines 1- 
2). In this process, each of the objects is sequentially processed as a respective candidate for 
segmentation into a respective current one of the object clusters in the series (see page 9, lines 
10-14; FIG. 3, step 174). For each of the candidate objects, a candidate object interval separating 
the candidate object from an adjacent object in the sequence already segmented into the current 
object cluster is determined (see page 9, lines 14-17). The candidate object interval is measured 
in the selected dimension of the context-related metadata (see page 9, lines 18-27). For each of 
the candidate objects, the candidate object interval is compared to a weighted measure of cluster 
extent for the current object cluster (see page 9, lines 14-17; FIG. 3, block 178). The measure of 
cluster extent corresponds to a current distance spanned by all the objects in the current object 
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cluster measured in the selected dimension of the context-related metadata (see page 9, lines 27- 
30). Exemplary embodiments in accordance with this process are shown in FIGS. 4A and 4B 
and described on page 9, line 31 - page 10, line 31, For each of the candidate objects, the 
candidate object interval is compared to a weighted measure of object density for the current 
object cluster (see page 10, line 32 - page 11, line 2; FIG. 3, block 208). The measure of object 
density corresponds to a measure of distribution of distances separating adjacent ones of the 
objects in the current object cluster measured in the selected dimension of the context-related 
metadata (see page 11, lines 2-11). An exemplary embodiment in accordance with this process 
is shown in FIG. 5 and described on page 11, line 12 - page 12, line 4). 

C. Dependent claims 2, 3, 5, 8, 10-14, 16, and 18 

Ckim 2 depends from claim 1 and recites that "the measure of cluster extent for each 
current object cluster corresponds to a temporal distance spanned by recorded generation times 
associated with all objects in the current object cluster." In some embodiments of the method of 
FIG. 3, the objects may be arranged in accordance with generation time meta data (see page 9, 
lines 21-22), and the extent of a cluster corresponds to the distance spanned by the objects 
contained within the cluster measured in the dimension (e.g., time or space) of the context- 
related meta data in accordance with which the object sequence is arranged (see page 9, lines 27- 
30). 

Claim 3 depends from claim 1 and recites that "the measure of cluster extent for each 
current object cluster corresponds to a spatial distance spanned by recorded generation locations 
associated with all objects in the current object cluster." In some embodiments of the method of 
FIG. 3, the objects may be arranged in accordance with generation location meta data (see page 
9, lines 23-24), and the extent of a cluster corresponds to the distance spanned by the objects 
contained within the cluster measured in the dimension (e.g., time or space) of the context- 
related meta data in accordance with which the object sequence is arranged (see page 9, lines 27- 
30). 

Claim 5 depends from claim 1 and recites that "the measure of object density for each 
current object cluster corresponds to an average spatial distance separating adjacent objects in 
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the current object cluster." On page 9, lines 24-25, the specification explains that the interval 
between successive objects corresponds to the geographic distance separating the locations 
where the objects were generated. On page 1 1, lines 4-11, the specification explains that in some 
implementations the object density corresponds to an average measure (e.g., a first order statistic, 
such as the arithmetic mean, the median, the mode, or the like, which describes or summarizes 
the distribution of successive ob ject intervals in the cluster) of the intervals separating objects in 
the cluster. 

Claim 8 depends from claim 1 and recites "applying to the measures of cluster extent 
respective weights that decrease with increasing sizes of the respective object clusters. 55 On page 
10, lines 4-5 and 21-23, the specification explains that, in some embodiments, the weighting 
factor (F) applied to the measures of cluster extent decreases with increasing cluster size. 

Claim 10 depends from claim 1 and recites "customizing at least one of the weights 
applied to the measures of cluster extent based on an analysis of objects in the corresponding 
object cluster." On page 14, lines 13-15, the specification explains that the weighting factor (F) 
applied to the measures of cluster extent "may be customized for a particular object generation 
environment based on an analysis of a collection of media objects captured in that environment," 
An example of this process is described on page 14, lines 16-28. 

Claim 1 1 depends from claim 10 and recites that "the customizing comprises scaling at 
least one of the weights applied to the measures of cluster extent based on a fractal dimension 
estimate of recorded time generation meta data associated with the objects in the collection." On 
page 14, lines 10-28, the specification explains that in some embodiments the fractal dimension 
of a user's media object capture behavior is estimated and used to scale one or more default 
ranges for the weight (F) that is applied to the measures of cluster extent. 

Claim 12 depends from claim 1 and recites "customizing at least one of the weights 
applied to the measures of cluster object density based on an analysis of objects in the 
corresponding object cluster." On page 14, lines 13-15, the specification explains that the 
weighting factor (G), which is applied to the measures of cluster object density, "may be 
customized for a particular object generation environment based on an analysis of a collection of 
media objects captured in that environment." 
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Claim 13 depends from claim 12 and recites that "the customizing comprises scaling at 
least one of the weights applied to the measures of cluster extent based on a fractal dimension 
estimate of recorded time generation meta data associated with the objects in the collection." On 
page 14, lines 10-28, the specification explains that in some embodiments the fractal dimension 
of a user's media object capture behavior is estimated and used to scale one or more default 
ranges for the weight (G) applied to the measures of cluster object density. 

Claim 14 depends from claim 1 and recites that "the processing further comprises 
comparing the object density of a candidate object cluster consisting of the current object cluster 
and the candidate object with the weighted measure of object density for the current object 
cluster." On page 12, lines 7-22, the specification explains that the segmentation engine 162 
compares the object density of a candidate cluster, consisting of the current cluster and the 
candidate object, with a weighted measure of the object density of the current cluster (see FIG. 3, 
step 225). 

Claim 16 depends from claim 14 and recites that "the measure of object density for each 
current object cluster corresponds to an average spatial distance separating adjacent objects in 
the current object cluster." On page 9, lines 24-25, the specification explains that the interval 
between successive objects corresponds to the geographic distance separating the locations 
where the objects were generated. On page 11, lines 4-11, the specification explains that in some 
implementations the object density corresponds to an average measure (e.g., a first order statistic, 
such as the arithmetic mean, the median, the mode, or the like, which describes or summarizes 
the distribution of successive object intervals in the cluster) of the intervals separating objects in 
the cluster. 

Claim 18 depends from claim 14 and recites that "the processing comprises determining 
the weighted measures of cluster extent by applying to the measures of cluster extent respective 
weights that decrease with increasing sizes of the respective object clusters." On page 10, lines 
21-23, the specification explains that, in some embodiments, the weighting factor (F) applied to 
the measures of cluster extent decreases with increasing cluster size. 
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D. Independent claim 22 

The aspect of the invention defined in independent claim 22 is a system of organizing a 
collection of objects arranged in a sequence ordered in accordance with a selected dimension of 
context-related metadata respectively associated with the objects. The system includes a 
segmentation engine that classifies the objects in the sequence to generate a series of object 
clusters. The segmentation engine sequentially processes each of the objects as a respective 
candidate for segmentation into a respective current one of the object clusters in the series. For 
each of the candidate objects, the segmentation engine determines a candidate object interval 
separating the candidate object from an adjacent object in the sequence already segmented into 
the current object cluster. The candidate object interval is measured in the selected dimension of 
the context-related metadata. For each of the candidate objects, the segmentation engine also 
compares the candidate object interval to a weighted measure of cluster extent for the current 
object cluster. The measure of cluster extent corresponds to a current distance spanned by all the 
objects in the current object cluster measured in the selected dimension of the context-related 
metadata. For each of the candidate objects, the segmentation engine additionally compares the 
candidate object interval to a weighted measure of cluster object density for the current object 
cluster. The measure of object density corresponds to a measure of distribution of distances 
separating. adjacent ones of the objects in the current object cluster measured in the selected 
dimension of the context-related metadata. 

FIG. 2 shows an embodiment of an object manager 12 in accordance with the aspect of 
the invention defined in claim 22. The object manager 12 organizes a collection of objects that 
are arranged into a sequence that is ordered in accordance with context-related meta data 
associated with the objects (see page 8, lines 16-17; FIG. 3). The context-related meta data may 
be, for example, meta data relating to the creation, termination, extent, operating environment, 
and other direct or circumstantial information), including one or both of time meta data and 
location (geographic or logical) meta data (see page 6, lines 9-12). 

The object manager 12 includes a segmentation engine 162 that classifies the objects in 
the sequence to generate a series of object clusters (see, e.g., page 5, lines 22-25, page 7, lines 
22-24, page 13, lines 1-2, and FIG. 2). In this process, the segmentation engine 162 sequentially 
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processes each of the objects as a respective candidate for segmentation into a respective current 
one of the object clusters in the series (see page 9, lines 10-14; FIG. 3, step 174). For each of the 
candidate objects, the segmentation engine 162 determines a candidate object interval separating 
the candidate object from an adjacent object in the sequence already segmented into the current 
object cluster (see page 9, lines 14-17). The candidate object interval is measured in the selected 
dimension of the context-related metadata (see page 9, lines 18-27). For each of the candidate 
objects, the segmentation engine 162 compares the candidate object interval to a weighted 
measure of cluster extent for the current object cluster (see page 9, lines 14-17; FIG. 3, block 
178). The measure of cluster extent corresponds to a current distance spanned by all the objects 
in the current object cluster measured in the selected dimension of the context-related metadata 
(see page 9, lines 27-30). Exemplary embodiments in accordance with this process are shown in 
FIGS. 4A and 4B and described on page 9, line 31 - page 10 5 line 31. For each of the candidate 
objects, the segmentation engine 162 compares the candidate object interval to a weighted 
measure of object density for the current object cluster (see page 10, line 32 - page 11, line 2; 
FIG. 3, block 208). The measure of object density corresponds to a measure of distribution of 
distances separating adjacent ones of the objects in the current object cluster measured in the 
selected dimension of the context-related metadata (see page 11, lines 2-11). An exemplary 
embodiment in accordance with this process is shown in FIG. 5 and described on page 11, line 
12 - page 12, line 4). 

E. Independent claim 23 

The aspect of the invention defined in independent claim 23 is a method of organizing a 
collection of objects. The objects from the collection are segmented into clusters. Context- 
related meta data is extracted. The context-related meta data is associated with the objects and is 
parsable into multiple levels of a name hierarchy. Names are assigned to clusters based on the 
extracted context-related meta data corresponding to a level of the name hierarchy selected to 
distinguish segmented clusters from one another. 

FIG. 6 shows an embodiment of a method of organizing a collection of objects in 
accordance with the aspect of the invention defined in claim 23 (see page 15, lines 7-8). In 
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accordance with this embodiment, objects from the collection axe segmented (see page 15, line 8; 
FIG. 6, step 230). The objects may be segmented in accordance with any object segmentation 
method, including one of the object segmentation methods described in the present application 
(see page 15, lines 8-10). Context-related meta data that is associated with the objects and is 
parsable into multiple levels of a name hierarchy is extracted (see page 15, lines 10-12; FIG. 6, 
step 232). In the exemplary embodiment shown in FIG. 7A, the context-related meta data 
corresponds to information relating to the times when the objects were generated (see page 15, 
lines 13-23). In the exemplary embodiment shown in FIG, 7B the context-related meta data 
corresponds to information relating to the locations where the objects were generated (see page 
15, line 24 - page 16, line 6). After the context-related meta data has been extracted (FIG. 6, step 
232), names are assigned to clusters based on the extracted context-related meta data 
corresponding to a level of the name hierarchy selected to distinguish segmented clusters from 
one another (see page 16, lines 7-10; FIG. 6, step 234). 

F. Dependent claim 29 

Claim 29 depends from claim 23 and recites "automatically naming objects in a given 
cluster based on the name assigned to the given cluster." On page 18, lines 1-7, the specification 
explains that in some embodiments of the method of FIG. 6 "The objects in any given cluster 
may be named automatically based on the name assigned to the given cluster." 

G. Independent claim 32 

The aspect of the invention defined in independent claim 32 is a system of organizing a 
collection of objects. The system includes a segmentation engine and a naming engine. The 
segmentation engine segments objects from the collection into clusters. The naming engine 
extracts context-related meta data associated with the objects and parsable into multiple levels of 
a name hierarchy. The naming engine also assigns names to each cluster based on the extracted 
context-related meta data corresponding to a level of the name hierarchy selected to distinguish 
segmented clusters from one another. 
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FIG. 2 shows an embodiment of an object manager 12 that organizes a collection of 
objects in accordance with the aspect of the invention defined in claim 32. The object manager 
12 includes a segmentation engine 162, a layout engine 164, and a naming engine 167 (see page 
7, line 13 - page 8, line 2). 

FIG. 6 shows an embodiment of a method that is executed by embodiments of the object 
manager 12 in accordance with the aspect of the invention defined in claim 23 (see page 15, lines 
4-8). These embodiments of the object manager 12 segment objects from the collection (see 
page 15, line 8; FIG. 6, step 230). The objects may be segmented in accordance with any object 
segmentation method, including one of the object segmentation methods described in the present 
application (see page 15, lines 8-10). These embodiments of the object manager 12 also extract 
context-related meta data that is associated with the objects and is parsable into multiple levels of 
a name hierarchy (see page 15, lines 10-12; FIG. 6, step 232). In the exemplary embodiment 
shown in FIG. 7A, the context-related meta data corresponds to information relating to the times 
when the objects were generated (see page 15, lines 13-23). In the exemplary embodiment 
shown in FIG. 7B the context-related meta data corresponds to information relating to the 
locations where the objects were generated (see page 15, line 24 - page 16, line 6). After the 
context-related meta data has been extracted (FIG. 6, step 232), these embodiments of the object 
manager 12 assign names to clusters based on the extracted context-related meta data 
corresponding to a level of the name hierarchy selected to distinguish segmented clusters from 
one another (see page 16, lines 7-10; Fig. 6, step 234). 

H. Independent claim 33 

The aspect of the invention defined in independent claim 33 is a method of organizing a 
collection of objects. In accordance with this method, a sequence of objects is accessed. The 
sequence is segmented into clusters each including multiple constituent objects arranged in a 
respective sequence in accordance with context-related meta data associated with the objects. 
For each object cluster, at least two constituent objects representative of beginning and ending 
instances in the corresponding object sequence are selected. In a user interface, the selected 
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representative objects of each cluster are graphically presented without graphically presenting 
representations of unselected ones of the constituent objects of the clusters. 



accordance with the aspect of the invention defined in claim 33. In accordance with this method, 
a sequence of objects is accessed (see page 18, lines 26-31). The sequence is segmented into 
clusters each including multiple constituent objects arranged in a respective sequence in 
accordance with context-related meta data associated with the objects. For each object cluster, at 
least two constituent objects representative of beginning and ending instances in the 
corresponding object sequence are selected (see page 18, lines 31-33). In the exemplary 
embodiment illustrated in FIG. 9, the layout engine 164 selects as representative objects the first 
and last objects 242, 244 and 246, 248 of a pair of neighboring clusters 250 and 252, 
respectively. In other implementations, the layout engine 164 selects a predetermined number of 
objects near each of the beginning and end of each cluster. In a user interface, the selected 
representative objects of each cluster are graphically presented without graphically presenting 
representations of unselected ones of the constituent objects of the clusters (see page 19, lines 5- 
7). In the exemplary embodiment illustrated in FIG. 9, the user interface 240 graphically 
presents the selected representative objects 242-248. 

I. Dependent claims 34, 36-39, 41-43, and 50 

Claim 34 depends from claim 33 and recites "graphically presenting a selected one of the 
clusters as a stack of partially overlapping images representative of multiple objects in the 
selected cluster." On page 14, lines 14-23, the specification explains that, "in some 
embodiments, a user may browse objects within a cluster through a user interface 280 that 
graphically presents an 'active cardstack' 282" (see FIG. 11). 

Claim 36 depends from claim 33 and recites "presenting the selected representative 
objects with the spacing between adjacent ones of the selected representative objects in the same 
cluster smaller than the spacing between adjacent ones of the selected representative objects in 
different clusters." On page 19, lines 6-7, the specification explains that "The representative 



FIG. 9 shows an embodiment of a method of organizing a collection of objects in 
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objects of any given cluster are presented closer to each other than to the representative objects 
of other clusters" (see FIG. 9). 

Claim 37 depends from claim 33 and recites "merging objects of one cluster into an 
adjacent cluster in response to user input." On page 19, lines 18-19, the specification explains 
that "The layout engine 164 is configured to merge objects of one cluster into another cluster in 
response to user input." 

Claim 38 depends from claim 37 and recites that "objects of one cluster are merged into 
an adjacent cluster in response to dragging and dropping of the objects to be merged." With 
reference to FIG. 9, the specification explains that in response to the user dragging and dropping 
representative object 244 of cluster 250 into cluster 252, layout engine 164 merges object 244 
into cluster 252 (see page 19, lines 18-23; FIG. 9). 

Claim 39 depends from claim 37 and recites that "the objects of the one cluster are 
merged into the adjacent cluster in response to user selection of an icon for merging the 
clusters." With reference to FIG. 9, the specification explains that "A user also may merge an 
entire cluster into another cluster by selecting one of the arrow icons 258, 260, 262, 164, which 
directs layout engine 164 to merge the cluster associated with the selected arrow icon into the 
neighboring cluster toward which the arrow icon is pointing" (see page 19, lines 23-29; FIG. 9). 

Claim 41 depends from claim 40 and recites that an "object distribution for a given 
cluster is presented as object instances plotted along an axis corresponding to a scaled 
representation of the context-related extent spanned by the given cluster." With reference to 
FIG. 10, the specification explains that "a user interface 270 displays histograms 272, 274 
representative of the distribution of objects in the clusters 250, 252" (see page 19, line 30 - page 
20, line 7; FIG. 10). The specification also explains that "Each histogram 272, 274 presents the 
number of object instances plotted along an axis corresponding to a scaled representation of the 
context-related extent (e.g., generation time in the example illustrated in FIG. 10) that is spanned 
by the corresponding cluster 250, 252" (see page 19, line 33 - page 20, line 2; FIG. 10). 

Claim 42 depends from claim 40 and recites "splitting a given cluster in response to user 
selection of a point in the representation of the object distribution presented for the given 
cluster." With reference to FIG. 10, the specification explains that "A user may direct layout 
engine 164 to split a given cluster 250, 252 into two groups by selecting a point in the 
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corresponding histogram 272, 274 between the two groups of objects that are to be split" (see 
page 20, lines 3-7; FIG. 10). 

Claim 43 depends from claim 40 and recites "automatically splitting a given cluster into 
two or more clusters in response to user input." With reference to FIG. 10, the specification 
explains that "A user may direct layout engine 164 to split a given cluster 250, 252 into two 
groups by selecting a point in the corresponding histogram 272, 274 between the two groups of 
objects that are to be split" (see page 20, lines 3-7; FIG. 10). 

Claim 50 depends from claim 33 and recites "graphically presenting at least one link to 
an object of a cluster arranged in a sequence in accordance with location-related meta data in a 
map format." With reference to FIG. 12B, the specification explains that in some embodiments 
in which objects are segmented based on location-related context information, a user may browse 
the objects with a user interface presenting links 312 arranged in a map format (see page 22, 
lines 16-20; FIG. 12B). 

J. Independent claim 5 1 

The aspect of the invention defined in independent claim 51 is a system of organizing a 
collection of objects. The system includes a user interface layout engine that accesses a 
sequence of objects from the collection segmented into clusters each including multiple objects 
arranged in a respective sequence in accordance with context-related meta data associated with 
the objects. The user interface layout engine selects for each object cluster at least two 
constituent objects representative of beginning and ending instances in the corresponding object 
sequence. In a user interface, the user interface layout engine graphically presents the selected 
representative objects of each cluster on a screen without graphically presenting representations 
of unselected ones of the constituent objects of the clusters. The user interface layout engine 
presents the selected representative objects with the spacing between adjacent ones of the 
selected representative objects in the same cluster smaller than the spacing between adjacent 
ones of the selected representative objects in different clusters. 

FIG. 2 shows an embodiment of an object manager that organizes a collection of objects 
in accordance with the aspect of the invention defined in claim 51 . The object manager 12 
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includes a layout engine 164 that cooperates with the user interface 166 to graphically present 
the selected representative objects of each cluster on a screen and allow a user to browse the 
segmented object collection (see page 7, lines 28-30). 

FIG. 9 shows an embodiment of a method that is executed by the layout engine 164 in 
accordance with the aspect of the invention defined in claim 33. In accordance with this method, 
the layout engine 164 segments a sequence of objects into clusters each including multiple 
constituent objects arranged in a respective sequence in accordance with context-related meta 
data associated with the objects is accessed (see page 18, lines 26-31). For each object cluster, 
the layout engine 164 selects at least two constituent objects representative of begimiing and 
ending instances in the corresponding object sequence (see page 1 8, lines 31-33). In the 
exemplary embodiment illustrated in FIG. 9, the layout engine 164 selects as representative 
objects the first and last objects 242, 244 and 246, 248 of a pair of neighboring clusters 250 and 
252, respectively. In other implementations, the layout engine 164 selects a predetermined 
number of objects near each of the beginning and end of each cluster. In a user interface, the 
layout engine 164 graphically presents the selected representative objects of each cluster without 
graphically presenting representations of unselected ones of the constituent objects of the clusters 
(see page 19, lines 5-7). In the exemplary embodiment illustrated in FIG. 9, the user interface 
240 graphically presents the selected representative objects 242-248. The representative objects 
of any given cluster are presented closer to each other than to the representative objects of other 
clusters (see page 19, lines 6-7; FIG. 9). 

VI. Grounds of Reiection to be Reviewed on Appeal 

A. Claims 1-32 and 51 stand rejected under 35 U.S.C. § 101 as being directed to non- 
statutory subject matter. 



B. 



Claims 1-51 stand rejected under 35 U.S.C. § 102(e) over Piatt (U.S. 



2003/0009469). 
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VII Argument 

A. Rejection of claims 1-32 and 51 under 35 U.S.C § 101 

The Examiner has rejected claims 1-32 and 51 under 35 U.S.C. § 101 as being directed to 
non-statutory subject matter. 

L Determining patentable subject matter under 35 U.S.C. § 101 

In accordance with the "Interim Guidelines for Examination of Patent Applications for 
Patent Subject Matter Eligibility" (published November 22, 2005; referred to herein as the 
"Interim Guidelines"), which clarifies the process described in MPEP § 2106 in view of the 
recent precedential Board decision in Ex parte Carl A. Lundgren , Appeal No. 2003-2088 (BPAI 
2005), the following procedure should be used for determining whether a claim is directed to 
patentable subject matter under 35 U.S.C. § 101: 



1. Determine whether the claim falls within one of the four enumerated categories: 
process, machine, manufacture, and composition of matter. 

2. Next, determine whether the claim is directed to nothing more than one of the 
following judicial exceptions: abstract idea (such as mathematical algorithms), 
natural phenomena, and laws of nature. In this regard, the issue is whether the 
claim would, in reality, preempt the use of a law of nature or abstract idea (e.g., a 
process that comprises every substantial practical application of an abstract idea). 
The Examiner must identify the abstraction, law of nature, or natural phenomenon 
and explain why the claim covers every substantial practical application thereof. 
If the Examiner meets this burden, she has established a prima facie case that the 
claims are not directed to patentable subject matter under 35 U.S.C. § 101. 

3. On the other hand, the claim is directed to patentable subject matter if it recites a 
practical application of any of the three judicial exceptions, which is established: 

a, if the claimed invention "transforms" an article or physical object to a 
different state or thing; or 
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b. if the claimed invention otherwise produces a useful, concrete, and 
tangible result. The result is useful if it is specific, substantial, credible, 
and specifically recited in the claim. The result is tangible if it is a "real- 
world result," as opposed to an abstract idea. The result is concrete if it 
can be assured (i.e., the process has a result that can be substantially 
repeatable or the claimed process must substantially produce the same 
result again.). 

4. If the claim is directed to a practical application of one of the judicial exceptions, 
producing a result tied to the physical world that does not preempt the judicial 
exception, then the claim meets the statutory requirement of 35 U.S.C. § 101. 

2. The Examiner's position and Applicants' rebuttal 

The Examiner has rejected claim 1-32 and 51 under 35 U.S.C. § 101 as being directed to 
non-statutory subject matter for the following reasons (see page 3, lines 1-6 of the final Office 
action): 

In claims 1-21, 23-31, the "method" is recited; however, all 
method claimed are abstract ideas not practical application, 
therefore it not limited to tangible, patent-eligible subject matter. 

In claims 22, 32 and 51, a "system of organizing" is recited; 
however, system claimed comprises only software components. 
However, it is a computer software per se. 

That is, the Examiner's position is that each of claims 1-32 and 51 falls within one of the 
four categories enumerated in 35 U.S.C. § 101, but the method claims 1-21 and 23-31 are 
directed to nothing more than abstract ideas and the system claims 22, 32, and 51 are directed to 
computer software and is therefore ineligible for patent protection "per se." 

With respect to the method claims 1-21 and 23-31, however, the Examiner has not 
identified the abstraction to which these claims are solely directed and has not explained why 
each of these claims covers every substantially practical application of that abstraction. 
Therefore, the Examiner has not established a prima facie case that these method claims are 
directed to non-statutory subject matter. 
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With respect to the system claims 22, 32, and 51, the Examiner's contention that the 
"system claimed comprises only software components" is not correct. Although the scope of 
these claims encompasses software components, these claims do not comprise "only software 
components." Moreover, there is no per se rule for computer-related inventions that obviates the 
need for the Examiner to establish a prima facie case of ineligibility for patent protection under 
35 U.S.C. § 101 (see, e.g., MPEP § 2106 and the Interim Guidelines). Since the Examiner has 
not given any proper reasons for the rejection of these system claims under 35 U.S.C. § 101, the 
Examiner has not established a prima facie case that these system claims are directed to non- 
statutory subject matter. 

For at least the reasons explained above, the Examiner's rejection of claims 1-32 and 51 
under 35 U.S.C. § 101 should be withdrawn. This rejection also should be withdrawn for the 
following additional reasons. 

a. Method claims 1 -2 1 

Independent claim 1 recites in part "classifying the objects in the sequence to generate a 
series of object clusters" (emphasis added). 

The "classifying" step recited in claim 1 does not involve merely solving mathematical 
problems or manipulating only numbers, abstract ideas or concepts (e.g., a bid or a bubble 
hierarchy) without some claimed practical application; nor do these steps consist solely of 
mathematical operations (i.e., converting one set of number into another set of numbers). 
Instead, in accordance with the ordinary and accustomed meaning of the term "classifying" (see, 
e.g., Merriam- Webster's Collegiate Dictionary, 10th Ed.), the claimed "classifying" step 
involves arranging the objects in classes or assigning the objects to a category "to generate a 
series of object clusters." In addition, whatever "abstract ideas" the Examiner believes claim 1 
manipulates, there is no reasonable way that claim 1 could be considered to cover every 
substantial practical application of those ideas. 

For at least these additional reasons, the Examiner's rejection of independent claim 1 
under 35 U.S.C. § 101 should be withdrawn. This rejection also should be withdrawn for the 
following additional reason. 
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Even if the Examiner can somehow establish that "classifying the objects in the sequence 
to generate a series of object clusters" involves nothing more than manipulation of abstract ideas, 
the invention defined in claim 1 produces a useful, concrete, and tangible result (i.e., the 
generation of a series of object clusters). As explained in the specification, the claimed result 
"may be used for any of a variety of applications, including permanent organization, collective 
annotation, and dynamic visualization" (page 5, lines 24-25). The result (i.e., a generated series 
of object clusters) is useful because it is specific, substantial, credible, and specifically recited in 
the claim. The result is tangible because it is a "real-world result" (i.e., a generated series of 
object clusters), as opposed to an abstract idea. The result is concrete because it can be assured 
(i.e., a given embodiment of the method defined in claim 1 will produce substantially the same 
result each time the method is executed; in other words, the method defined in claim 1 has a 
result that is substantially reproducible). 

The Board's attention is drawn to the Federal Circuit's decision in AT&T Corp. v. Excel 
Communications, Inc. , 172 F.3d 1352 (Fed. Cir. 1999), which provides guidance in the 
interpretation of the Interim Guidelines (see, e.g., page 1 of the Interim Guidelines). In this 
decision, the Federal Circtui held that (emphasis added): 

As previously explained, AT&T's claimed process employs 
subscribers' and call recipients' PICs as data, applies Boolean 
algebra to those data to determine the value of the PIC indicator, 
and applies that value through switching and recording 
mechanisms to create a signal useful for billing purposes. In State 
Street, we held that the processing system there was patentable 
subject matter because the system takes data representing discrete 
dollar amounts through a series of mathematical calculations to 
determine a final share price - a useful, concrete, and tangible 
result See 149 F.3d at 1373, 47 USPQ2d at 1601. 

In this case, Excel argues, correctly, that the PIC indicator value is 
derived using a simple mathematical principle (p and q). But that is 
not determinative because AT&T does not claim the Boolean 
principle as such or attempt to forestall its use in any other 
application. It is clear from the written description of the '184 
patent that AT&T is only claiming a process that uses the Boolean 
principle in order to determine the value of the PIC indicator. The 
PIC indicator represents information about the call recipient's PIC, 
a useful non-abstract result that facilitates differential billing of 
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long-distance calls made by an IXC's subscriber. Because the 
claimed process applies the Boolean principle to produce a useful 
concrete, tangible result without pre-empting other uses of the 
mathematical principle, on its face the claimed process 
comfortably falls within the scope of 101. See Arrhythmia 
Research Tech. Inc. v. Corazonix Corp., 958 F,2d 1053, 1060, 22 
USPQ2d 1033, 1039 (Fed. Cir. 1992) ("That the product is 
numerical is not a criterion of whether the claim is directed to 
statutory subject matter."). 

Thus, claim 1 of AT&T's ; 184 patent, which was the subject of the above-mentioned 
Federal Circuit decision, was held to be directed to patentable subject matter for reciting the 
generation of a message record that includes a PIC representing information about the call 
recipient's PIC. The classifying of objects "to generate a series of object clusters" of the type 
defined in claim 1 of the present application renders the invention defined by the claim 
patentable subject matter under 35 U.S.C. § 101 for the same reasons. 

For at least this additional reason, the Examiner's rejection of independent claim 1 under 
35 U.S.C. § 101 should be withdrawn. 

Each of claims 2-22 incorporates the features of independent claim 1 and therefore is 
directed to patentable subject matter for at least the same reasons. 

b. Method claims 23-3 1 

Independent claim 23 recites: 

23. A method of organizing a collection of objects, 
comprising: 

segmenting objects from the collection into clusters; 

extracting context-related meta data associated with the 
objects and parsable into multiple levels of a name hierarchy; and 

assigning names to clusters based on the extracted context- 
related meta data corresponding to a level of the name hierarchy 
selected to distinguish segmented clusters from one another. 

The "segmenting," "extracting," and "assigning" steps recited in claim 23 do not involve 
merely solving mathematical problems or manipulating only numbers, abstract ideas or concepts 
(e.g., a bid or a bubble hierarchy) without some claimed practical application; nor do these steps 
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consist solely of mathematical operations (i.e., converting one set of number into another set of 
numbers). Instead, in accordance with the ordinary and accustomed meaning of the terms 
"segmenting/' "extracting," and "assigning" (see, e.g., Merriam-Webster's Collegiate 
Dictionary, 10th Ed.), the claimed "segmenting," "extracting," and "assigning" steps involves 
dividing objects from a collection into clusters, determining context-related meta data associated 
with the objects, and associating names to the clusters. In addition, whatever "abstract ideas" the 
Examiner believes claim 23 manipulates, there is no reasonable way that claim 23 could be 
considered to cover every substantial practical application of those ideas. 

For at least these additional reasons, the Examiner's rejection of independent claim 23 
under 35 U.S.C. § 101 should be withdrawn. This rejection also should be withdrawn for the 
following additional reason. 

Even if the Examiner can somehow establish that the "segmenting," "extracting," and 
"assigning" steps recited in claim 23 involve nothing more than manipulation of abstract ideas, 
the invention defined in claim 23 produces a useful, concrete, and tangible result (i.e., object 
clusters with assigned names). As explained in the specification, the claimed result provides 
"meaningful, context-sensitive names to the object clusters" (page 6, lines 1-3). The result is 
useful because it is specific, substantial, credible, and specifically recited in the claim. The result 
is tangible because it is a "real-world result" (i.e., object clusters with assigned names), as 
opposed to an abstract idea. The result is concrete because it can be assured (i.e., a given 
embodiment of the method defined in claim 23 will produce substantially the same result each 
time the method is executed; in other words, the method defined in claim 23 has a result that is 
substantially reproducible). 

For at least this additional reason, the Examiner's rejection of independent claim 23 
under 35 U.S.C. § 101 should be withdrawn. 

Each of claims 24-3 1 incorporates the features of independent claim 1 and therefore is 
directed to patentable subject matter for at least the same reasons. 
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c. System claims 22, 32, and 5 1 

None of the system claims 22, 32 and 51 recites (i) nonfunctional descriptive material, 
such as music, literary works and a compilation or mere arrangement of ideas" (see MPEP § 
2106 IV.B.l), (ii) functional descriptive material (e.g., data structures or computer listings) not 
claimed as embodied in computer-readable material (see MPEP § 2106 IV.B.l (a)), or (iii) 
"nothing but the physical characteristics of a form of energy, such as a frequency, voltage, or the 
strength of a magnetic field" (see MPEP § 2106 IV.B.l(c)), Instead, each of these claims is an 
apparatus claim that recites one or more "engines" that are operable to perform specified 
functions. Accordingly, none of these claims is directed to non-statutory subject matter per se, 
contrary to the Examiner's position (see MPEP § 2106). 

In addition, none of the system claims 22, 32, and 51 involves merely solving 
mathematical problems or manipulating only numbers, abstract ideas or concepts (e.g., a bid or a 
bubble hierarchy) without some claimed practical application; nor do these systems consist 
solely of mathematical operations (i.e., converting one set of number into another set of 
numbers). Instead, each of the system claims 22, 32, and 51 recites more than one of the three 
judicial exceptions, including recitation of a practice application: 

• Regarding claim 22, the claimed "segmentation engine" is operable to classify the 
objects "to generate a series of object clusters" (emphasis added). 

• Regarding claim 32, the claimed "segmentation engine" and "naming engine" are 
operable to divide objects from a collection into clusters, extract context-related meta 
data associated with the objects, and assign names to the clusters. 

• Regarding claim 51, the claimed "layout engine" is operable to graphically present 
the selected representative objects of each cluster in a user interface . 

Moreover, whatever "abstract ideas" the Examiner believes claims 22, 32, and 51 respectively 
manipulate, there is no reasonable way that these claims could be considered to cover every 
substantial practical application of those ideas. 

In addition, the functional limitations recited in the system claims 22 and 32 essentially 
track the pertinent method steps respectively recited in method claims 1 and 23. Therefore, even 
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if the Examiner can somehow establish that the functional features recited in the system claims 
22 and 32 involve nothing more than manipulation of abstract ideas, the aspects of the invention 
defined in these system claims produce useful, concrete, and tangible results for the same reasons 
explained above in connection with the method claims 1 and 23. 

The functional limitations recited in the system claim 51 essentially track the pertinent 
method steps recited in method claim 33. Therefore, even if the Examiner can somehow 
establish that the functional features recited in the system claim 51 involve nothing more than 
manipulation of abstract ideas, the aspects of the invention defined in this system claim produce 
a useful, concrete, and tangible result for the same reason as independent method claim 33, 
which the Examiner has found to be directed to patentable subject matter. The result produced 
by the subject matter defined in independent system claim 51 is at least as "tangible" as the result 
produced by the subject matter of independent method claim 33. Therefore, there is no 
reasonable basis for the Examiner's different treatment of these claims under 35 U.S.C. § 101 . 

For at least these additional reasons, the Examiner's rejection of the system claims 22, 
32, and 51 under 35 U.S.C. § 101 should be withdrawn. 

B. Rejection of claims 1-51 under 35 U.S.C. § 102(e) over Piatt (U.S. 



The Examiner has rejected claims 1-51 under 35 U.S.C. § 102(e) over Piatt (U.S. 
2003/0009469). 

1 . Overview of Piatt' s disclosure 

Piatt discloses a cataloging method and three clustering methods for indexing media 
objects in a collection. 

Regarding the cataloging method, Piatt teaches that "The cataloging method according to 
the present invention essentially involves four phases: data collection, data correlation, data 
refinement and data compaction" fl[ 39). In accordance with Piatt's teachings: 



2003/0009469) 
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• "Data collection includes gathering ancillary data before, during or after the 
capture of a media object" (| 40). 

• "Data correlation is the mechanism by which one or more pieces of data from the 
data collection phase are integrated and synthesized to form new data that may be 
used later to identify, organize, and search media objects" flj 42). 

• "Data refinement relates to the process of managing and extending the data 
correlation process to find more meaningful data and to reject false correlations" 
(11 43). 

• "Data compaction relates to the process of removing redundant and/or useless 
data" (t 44). 

The first clustering method is shown in FIG. 9 and described in paragraphs 59-63. In 
accordance with this method, images are sorted and processed in order of creation time. The 
creation time of each image is compared to a lasttime variable (see 1 59; FIG. 9, step S904). If 
the difference between the creation time of the current image and the lasttime variable is less 
than X hours, the image is added to the current collection and the lasttime variable is set equal to 
the creation time of the current image (see |1f 59 and 60; FIG. 9, steps S905 and S906); 
otherwise, the current image is added to a new empty current collection (see ^ 59; FIG. 9, step 
S910). The variable X may be set to a default value or a value set by the user (see f 59). 
Alternatively, the value of X may be set adaptively in accordance with equation (1), which 
computes a running average of the log time gaps between adjacent ones of the sorted 
photographs (see equation (1) and 1} 63). 

The second clustering method is shown in FIG. 10 and described in paragraphs 64-74. In 
accordance with this method, images are sorted by creation time (see ^ 67; FIG. 10, step SI 003). 
The sorted images then are clustered based on the pairwise difference between color histograms 
of adjacent pairs of the images in accordance with Omohundrp's best-first model merging 
technique (see f 72; FIG. 10, steps S1005, S1006, and S1010-S1013). According to Piatt flj 72): 

. . . The technique starts with each item in its own cluster. Then, at 
every step, the distance between every pair of adjacent clusters is 
computed. The pair with the smallest distance is merged together 
to yield a new cluster. This new cluster replaces the two, old 
clusters in the sequence, and the clustering continues until the 
desired number of clusters is reached. . . . 
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The third clustering method is shown in FIG. 1 1 and described in paragraph 75. This 
"method combines temporal and color clustering of photographs" (f 75 , lines 2-3). In this 
method, the images first are clustered in accordance with the first (time-based) clustering method 
shown in FIG. 9 (see ]j 75; FIG. 1 1 ? step SI 102). The resulting clusters "are scanned in steps 
SI 103-S1 107 to check for failure of the temporal clustering" (see 1 75, lines 9-1 1). Overly large 
clusters are identified by comparing the number of images in the clusters to a threshold (see ^ 75, 
lines 11-12). If the number is greater than the threshold for a current cluster, the current cluster 
is split in to sub-clusters in accordance with the second (color-based) clustering algorithm shown 
in FIG. 10 (see 175, lines 12-25). 

2. Independent claim 1 



Claim 1 recites: 

1 . A method of organizing a collection of objects arranged in 
a sequence ordered in accordance with a selected dimension of 
context-related metadata respectively associated with the objects, 
comprising: 

classifying the objects in the sequence to generate a series 
of object clusters, wherein the classifying comprises sequentially 
processing each of the objects as a respective candidate for 
segmentation into a respective current one of the object clusters in 
the series and, for each of the candidate objects, 

determining a candidate object interval separating the 
candidate object from an adjacent object in the 
sequence already segmented into the current object 
cluster, the candidate object interval being 
measured in the selected dimension of the context- 
related metadata, 

comparing the candidate object interval to a weighted 
measure of cluster extent for the current object 
cluster, the measure of cluster extent corresponding 
to a current distance spanned by all the objects in 
the current object cluster measured in the selected 
dimension of the context-related metadata, and 
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comparing the candidate object interval to a weighted 
measure of object density for the current object 
cluster, the measure of object density corresponding 
to a measure of distribution of distances separating 
adjacent ones of the objects in the current object 
cluster measured in the selected dimension of the 
context-related metadata. 



The Examiner's rejection of claim 1 under 35 U.S.C, § 102(e) over Piatt should be 
withdrawn because Piatt does not teach "comparing the candidate object interval to a weighted 
measure of cluster extent for the current object cluster, the measure of cluster extent 
corresponding to a current distance spanned by all the objects in the current object cluster 
measured in the selected dimension of the context-related metadata," as recited in claim 1 . This 
feature of claim 1 is referred to herein as "the cluster extent based feature of claim 1." 

3, The Examiner's position and Appellant's rebuttal 

a. The Examiner's position 

In support of the rejection of claim 1, the Examiner has stated that (final Office action, 
page 4, line 14 - page 5, line 5): 



For claim 1, Piatt teaches . . . 

"comparing the candidate object interval to a weighted measure of 
cluster extent for the current object cluster, the measure of the 
cluster extent corresponding to a current distance spanned by all 
the object in the current object cluster measured in the selected 
dimension of the content-related metadata". (See paragraph 
0037,0052-0055, 0063-0072]), "comparing the candidate object 
interval to a weighted measure of cluster extent for the current 
object cluster, the measure of the cluster extent corresponding to a 
current distance spanned by all the object in the current object 
cluster measured in the selected dimension of the content-related 
metadata" (See paragraph [0063-0072]) and comparing the 
candidate object interval to a weighted measure of object density 
for the current object cluster, the measure of the object density 
corresponding to a measure of the distribution of distances 
separating adjacent ones of the objects in the current object cluster 
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measured in the selected dimension of the context-related 
metadata. (See paragraph [0063-0072]). 

b. Appellant's rebuttal: The Examiner has not shown that Piatt teaches each and 
every aspect of the invention defined in claim 1 

In the rationale quote above, the Examiner appears to have impermissibly relied on a 
single element of Piatt's disclosure to meet two separate and distinct elements of claim 20, 
effectively reading one of these claim elements out of the claim. In particular, the Examiner 
appears to have relied on the same adaptive threshold X (defined in equation (1) and described in 
1 63) to meet both the "weighted measure of cluster extent" and the "measure of object density" 
that are recited in claim 1 . For example, the Examiner has indicated that the merging of 
consecutive candidate objects based on the weighted measure of cluster extent in accordance 
with dependent claim 6 is disclosed by Piatt in paragraphs 60-63, which describe the first (time- 
based) clustering method that uses the adaptive threshold X. The Examiner also has stated that 
"Piatt teaches 'wherein the measure of object density for each object cluster corresponds to a 
moving average distance separating adjacent objects in the current object cluster' (See paragraph 
[0059-0060])" (see page 9, third paragraph, of the final Office action). Paragraphs 59-60, 
however, also describe the first (time-based) clustering method that uses the same adaptive 
threshold X. 

Without a specific indication of which elements of Piatt's disclosure the Examiner 
believes respectively correspond to the "weighted measure of cluster extent" and the "measure of 
object density" that are recited in claim 1, it appears that the Examiner has failed to establish a 
prima facie case of anticipation of claim 1 under 35 U.S.C. § 102(e) (see, e.g., MPEP § 
706.02(a): "for anticipation under 35 U.S.C. 102, the reference must teach every aspect of the 
claimed invention either explicitly or impliedly"). 

In addition, as explained in detail below, none of the cited paragraphs (i.e., Xi 37, 52-55, 
and 63-72) teaches the cluster extent based feature of claim 1 . 
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Paragraph 37 



FIG. 7 in which 



In paragraph 37, Piatt teaches that media objects may be cataloged based upon correlation 
of the data for the media object and/or data describing the media object (metadata, such as data, 
time, location) with known data and metadata. This disclosure does not teach "comparing the 
candidate object interval to a weighted measure of cluster extent for the current object cluster, 
the measure of cluster extent corresponding to a current distance spanned by all the objects in the 
current object cluster measured in the selected dimension of the context-related metadata/' as 
recited in claim 1 . 

ii. Paragraphs 52-54 

In paragraphs 52-54, Piatt discloses aspects of the correlation-based cataloging method 
described above in § VTI.B.l. 

In paragraph 52, Piatt describes steps S700-S706 of the method shown in 

i 

"the date and location may be correlated to organize images." In accordance with this method, if 
the date associated with an image is within a date range corresponding to the user's vacation, 
then all objects having the same date are added to a pruned list (see f 52; FIG. 7, steps S700- 
S706). Contrary to the Examiner's statement, this disclosure does not teach "comparing the 
candidate object interval to a weighted measure of cluster extent for the current object cluster," 
as recited in claim 1. Indeed, the date that is associated with an image is not "a (Candidate object 
interval separating the candidate object from an adjacent object in the sequence already 
segmented into the current object cluster." In addition, the fixed date range corresponding to the 
user's vacation does not constitute a weighted measure of cluster extent for the cjurrent object 
cluster, where "the measure of cluster extent corresponding to a current distance ispanned by all 
the objects in the current obiect cluster measured in the selected dimension of the context-related 
metadata," as recited in claim 1 (emphasis added). 

In paragraph 53, Piatt describes steps S713-S720 of the method shown in; FIG. 7. In 
accordance with this method, if the location associated with an image is within 5f) miles of the 
vacation location, then all of the images at the same location are added to the pruned list (see 1} 
53, FIG. 7, steps 713-S716 and S706). Otherwise, other cataloging processes are performed 
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for the current 
associated with 
an adjacent 

FIG. 7, Instep 



(e.g., add to the pruned list images having features in common with the selected image; see FIG. 
7, steps S717 and S718). Contrary to the Examiner's statement, this disclosure does not teach 
"comparing the candidate object interval to a weighted measure of cluster extent 
object cluster," as recited in claim 1. Indeed, the location or other feature that is 
an image is not "a candidate object interval separating the candidate object from 
object in the sequence already segmented into the current object cluster." 

In paragraph 54, Piatt describes steps S707-S712 of the method shown in ] 
S707, "the images in the pruned list are processed to determine those images having common 
features with the image to be catalogued." In step S708, "an inference is made on how the image 

should be catalogued based upon the information obtained in step S707." Contrary to the 

| 

Examiner's statement, this disclosure does not teach "comparing the candidate object interval to 

I 

a weighted measure of cluster extent for the current object cluster," as recited in blaim 1. Indeed, 

Piatt does not teach anything that would have led on skilled in the art to believe that the inference 

i 

is made based on a comparison of "a candidate object interval separating the candidate object 
from an adjacent object in the sequence already segmented into the current object cluster" with a 
weighted measure of cluster extent for the current object cluster, as defined in claim 1 . 

i 

iii. Paragraph 55 j 

In paragraph 55, Piatt teaches that a search for images taken at Grandparent's house may 
be performed by going through all of the images and determining which images have GPS 
locations corresponding to the location of Grandparents house or by determining! which photos 
were taken during a trip to the Grandparent's house listed in a local calendar. Contrary to the 
Examiner's statement, this disclosure does not teach "comparing the candidate object interval to 
a weighted measure of cluster extent for the current object cluster," as recited in claim 1. 

iv. Paragraph 63 

In paragraph 63, Piatt describes equation (1), which is used to compute the adaptive 

threshold X for the first (time-based) clustering method described above in § VII.B.l. In 

I 

accordance with the first clustering method, if the difference between the creation time of the 
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current image and the lasttime variable is less than X hours, the image is added to the current 
collection and the lasttime variable is set equal to the creation time of the current! image (see ^ 
59 and 60; FIG. 9, steps S905 and S906); otherwise, the current image is added tp a new empty 
current collection (see ^ 59; FIG. 9, step S910). The adaptive threshold X, however, is not a 
weighted measure of cluster extent for the current object cluster, where the measjire of cluster 
extent corresponds to a current distance spanned by all the objects in the current 'object cluster 

j 

measured in the selected dimension of the context-related metadata, as defined in claim 1 . 

' i 

Instead, the value of X depends on the running average of the log time gaps between adjacent 
photographs (see ^ 63, lines 4-6). 



v. Paragraphs 64-72 



ustering method 



all of the 



In paragraphs 64-72, Piatt discloses aspects of the second (color-based) c. 
shown in FIG, 10 and described above in § VIIJ3.1. 

In paragraph 64, Piatt describes step SI 001 in which color histograms of all of the images 

are created. j 

i 

In paragraph 65, Piatt describes how the color histograms may be estimated. 
In paragraph 66, Piatt describes step S 1002 in which the creation times of 
images are extracted. 

In paragraph 67, Piatt describes step SI 003 in which all the images are sorted by creation 

time. 

In paragraph 68, Piatt describes step SI 004 in which the pairwise distances between the 
color histograms of adjacent pairs of ordered images are computed. 

In paragraph 69, Piatt describes aspects of equations (2) and (3). 

In paragraph 70, Piatt describes how the distance between two images or i 
determined. 

In paragraph 71, Piatt discloses that distance metric other than the one defined in 
equation (4) may be used in step SI 004. 

In paragraph 72, Piatt discloses that the sorted images then are clustered based on the 
pairwise difference between color histograms of adjacent pairs of the images in accordance with 



clusters can be 
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Omohundro's best-first model merging technique (see ^ 72; FIG. 10, steps S1005, S1006, and 
S1010-S1013). According to Piatt (If 72): 



. . . The technique starts with each item in its own cluster. Then, at 
every step, the distance between every pair of adjacent clusters is 
computed. The pair with the smallest distance is merged together 
to yield a new cluster. This new cluster replaces the two old 
clusters in the sequence, and the clustering continues until the 
desired number of clusters is reached. . . . 



Thus, contrary to the Examiner's statement, Piatt's disclosure in paragraphs 64-72 does 
not teach "comparing the candidate object interval to a weighted measure of cluster extent for the 
current object cluster." Indeed, in accordance with Piatt's teachings the distances between 
adjacent clusters are compared to each other, not to a weighted measure of cluster extent, where 
"the measure of cluster extent corresponding to a current distance spanned by all the objects in 
the current object cluster measured in the selected dimension of the context-related metadata," as 
defined in claim 1 . 

c. Conclusion 

For the reasons explained above, the Examiner's rejection of independent claim 1 under 
35 U.S.C. § 102(e) over Piatt should be withdrawn. 

4, Dependent claims 2-21 

Each of claims 2-21 incorporates the features of independent claim 1 and therefore is 
patentable over Piatt for at least the same reasons explained above. Claims 2, 3, 5, 8, 10-14, 16, 
and 18 also are patentable over Piatt for the following additional reasons. 

a. Claim 2 

Claim 2 recites that "the measure of cluster extent for each current object cluster 
corresponds to a temporal distance spanned by recorded generation times associated with all 
objects in the current object cluster." The Examiner has stated that Piatt discloses this feature in 
paragraphs 12, 37, 41, 45, 58, and 59. None of the cited paragraphs, however, supports the 
Examiner's statement. 



Applicant : Ullas Gargi 
Serial No. : 10/631,369 
Filed : July 31 ,2003 



Attorney's Docket No.: 200207387-1 
Appeal Brief dated Jan. 17, 2007 
Reply to final action dated Aug. 23, 2006 



Page : 31 of 71 

In paragraph 12, Piatt teaches that "Media objects may be clustered based on time-based 
clustering, content-based clustering or based on a combination of both." This disclosure, 
however, does not teach that "the measure of cluster extent for each current object cluster 
corresponds to a temporal distance spanned by recorded generation times associated with all 
objects in the current object cluster." 

In paragraph 37, Piatt teaches that media objects may be cataloged based upon correlation 
of the data for the media object and/or data describing the media object (metadata, such as data, 
time, location) with known data and metadata. This disclosure, however, does not teach that "the 
measure of cluster extent for each current object cluster corresponds to a temporal distance 
spanned by recorded generation times associated with all objects in the current object cluster." 

In paragraph 41, Piatt teaches that during the data collection phase of the cataloging 
method data, such as the date and time of a photograph being taken and sampled audio, may be 
gathered. This disclosure, however, does not teach that "the measure of cluster extent for each 
current object cluster corresponds to a temporal distance spanned by recorded generation times 
associated with all objects in the current object cluster." 

In paragraph 45, Piatt teaches that the cataloging process may include processing the 
media object with respect to time, date, and/or location, comparing the relevant data with 
threshold data, or performing an inexact search. This disclosure, however, does not teach that 
"the measure of cluster extent for each current object cluster corresponds to a temporal distance 
spanned by recorded generation times associated with all objects in the current object cluster." 

In paragraph 58, Piatt provides an overview of the first (time-based) clustering method. 
This disclosure, however, does not teach that "the measure of cluster extent for each current 
object cluster corresponds to a temporal distance spanned by recorded generation times 
associated with all objects in the current object cluster." 

In paragraph 59, Piatt describes the first (time-based) clustering method. In accordance 
with this method, images are sorted and processed in order of creation time. The creation time of 
each image is compared to a lasttime variable (see | 59; FIG. 9, step S904). If the difference 
between the creation time of the current image and the lasttime variable is less than X hours, the 
image is added to the current collection and the lasttime variable is set equal to the creation time 
of the current image (see Ifff 59 and 60; FIG. 9, steps S905 and S906); otherwise, the current 
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image is added to a new empty current collection (see U 59; FIG. 9, step S910). The variable X 
may be set to a default value or a value set by the user (see f 59). In either of these cases, the 
variable X does not correspond to a temporal distance spanned by recorded generation times 
associated with all objects in the current object cluster. Alternatively, the value of X may be set 
adaptively in accordance with equation (1), which computes a running average of the log time 
gaps between adjacent ones of the sorted photographs (see equation 1 and K 63). In this case, the 
variable X does not correspond to a temporal distance spanned by recorded generation times 
associated with all objects in the current object cluster. Instead, the value of X depends on the 
running average of the log time gaps between adjacent photographs (see 1 63, lines 4-6). 

For these additional reasons, the Examiner's rejection of claim 2 under 35 U.S.C. § 
102(e) over Piatt should be withdrawn. 

b. Claim 3 

Claim 3 recites that "the measure of cluster extent for each current object cluster 
corresponds to a spatial distance spanned by recorded generation locations associated with all 
objects in the current object cluster." The Examiner has stated that Piatt discloses this feature in 
paragraphs 34, 37, 45, 46, and 52-55. None of the cited paragraphs, however, supports the 
Examiner's statement. 

In paragraph 34, Piatt defines the terms "cataloging" and "indexing" and describes the 
types of data that may be used for indexing. This disclosure, however, does not teach that "the 
measure of cluster extent for each current object cluster corresponds to a spatial distance spanned 
by recorded generation locations associated with all objects in the current object cluster." 

In paragraph 37, Piatt defines the terms "cataloging" and "indexing" and describes the 
types of data that may be used for indexing. This disclosure, however, does not teach that "the 
measure of cluster extent for each current object cluster corresponds to a spatial distance spanned 
by recorded generation locations associated with all objects in the current object cluster." 

In paragraph 45, Piatt teaches that the cataloging process may include processing the 
media object with respect to time, date, and/or location, comparing the relevant data with 
threshold data, or performing an inexact search. This disclosure, however, does not teach that 
"the measure of cluster extent for each current object cluster corresponds to a spatial distance 
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spanned by recorded generation locations associated with all objects in the current object 
cluster." 

In paragraph 46, Piatt describes a process of using a global calendar to catalog a media 
object. In this process, date information for an image to be cataloged is compared to dates in the 
global calendar. If the date corresponds to January 1 st , it is inferred that the image was taken on 
New Year's Day and this inference may be used to catalog the image . This disclosure, however, 
does not teach that "the measure of cluster extent for each current object cluster corresponds to a 
spatial distance spanned by recorded generation locations associated with all objects in the 
current object cluster." 

In paragraphs 52-54, Piatt discloses aspects of the correlation-based cataloging method 
described above in § VII.B.l. 

In paragraph 52, Piatt describes steps S700-S706 of the method shown in FIG. 7 in which 
"the date and location may be correlated to organize images." In accordance with this method, if 
the date associated with an image is within a date range corresponding to the user's vacation, 
then all objects having the same date are added to a pruned list (see ^ 52; FIG. 7, steps S700- 
S706). Contrary to the Examiner's statement, this disclosure does not teach that "the measure of 
cluster extent for each current object cluster corresponds to a spatial distance spanned by 
recorded generation locations associated with all objects in the current object cluster." 

In paragraph 53, Piatt describes steps S713-S720 of the method shown in FIG. 7. In 
accordance with this method, if the location associated with an image is within 50 miles of the 
vacation location, then all of the images at the same location are added to the pruned list (see If 
53, FIG. 7, steps 713-S716 and S706). Otherwise, other cataloging processes are performed 
(e.g., add to the pruned list images having features in common with the selected image; see FIG. 
7, steps S717 and S718). Contrary to the Examiner's statement, this disclosure does not teach 
that "the measure of cluster extent for each current object cluster corresponds to a spatial 
distance spanned by recorded generation locations associated with all objects in the current 
object cluster." Indeed, the 50-mile range used in the example described in paragraph 53 does 
not correspond to "a spatial distance spanned by recorded generation locations associated with 
all objects in the current object cluster." Furthermore, the location or other feature that is 
associated with an image is not "a candidate object interval separating the candidate object from 
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an adjacent object in the sequence already segmented into the current object cluster," which is 
compared to the measure of cluster extent in accordance with claim 1. 

In paragraph 54, Piatt describes steps S707-S712 of the method shown in FIG. 7. In step 
S707, "the images in the pruned list are processed to determine those images having common 
features with the image to be catalogued." In step S708, "an inference is made on how the image 
should be catalogued based upon the information obtained in step S707." Contrary to the 
Examiner's statement, this disclosure does not teach that "the measure of cluster extent for each 
current object cluster corresponds to a spatial distance spanned by recorded generation locations 
associated with all objects in the current object cluster." Indeed, Piatt does not teach anything 
that would have led on skilled in the art to believe that the inference is made based on a 
comparison of "a candidate object interval separating the candidate object from an adjacent 
object in the sequence already segmented into the current object cluster" with a weighted 
measure of cluster extent that "corresponds to a spatial distance spanned by recorded generation 
locations associated with all objects in the current object cluster," as recited in claim 3. 

In paragraph 55, Piatt teaches that a search for images taken at Grandparent's house may 
be performed by going through all of the images and determining which images have GPS 
locations corresponding to the location of Grandparents house or by determining which photos 
were taken during a trip to the Grandparent's house listed in a local calendar. Contrary to the 
Examiner's statement, this disclosure does not teach that "the measure of cluster extent for each 
current object cluster corresponds to a spatial distance spanned by recorded generation locations 
associated with all objects in the current object cluster." 

For these additional reasons, the Examiner's rejection of claim 3 under 35 U.S.C. § 
102(e) over Piatt should be withdrawn. 

c. Claim 5 

Claim 5 recites that cc the measure of object density for each current object cluster 
corresponds to an average spatial distance separating adjacent objects in the current object 
cluster." The Examiner has stated that Piatt discloses this feature in paragraphs 45, 46, and 53. 
None of the cited paragraphs, however, supports the Examiner's statement. 
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In paragraph 45, Piatt teaches that the cataloging process may include processing the 
media object with respect to time, date, and/or location, comparing the relevant data with 
threshold data, or performing an inexact search. This disclosure, however, does not teach that 
"the measure of object density for each current object cluster corresponds to an average spatial 
distance separating adjacent objects in the current object cluster." 

In paragraph 46, Piatt describes a process of using a global calendar to catalog a media 
object. In this process, date information for an image to be cataloged is compared to dates in the 
global calendar. If the date corresponds to January 1st, it is inferred that the image was taken on 
New Year's Day and this inference may be used to catalog the image. This disclosure, however, 
does not teach that "the measure of object density for each current object cluster corresponds to 
an average spatial distance separating adjacent objects in the current object cluster." 

In paragraph 53, Piatt describes steps S713-S720 of the method shown in FIG. 7. In 
accordance with this method, if the location associated with an image is within 50 miles of the 
vacation location, then all of the images at the same location are added to the pruned list (see f 
53, FIG. 7, steps 713-S716 and S706). Otherwise, other cataloging processes are performed 
(e.g., add to the pruned list images having features in common with the selected image; see FIG. 
7, steps S717 and S718). Contrary to the Examiner's statement, this disclosure does not teach 
that "the measure of object density for each current object cluster corresponds to an average 
spatial distance separating adjacent objects in the current object cluster." Indeed, the 50-mile 
range used in the example described in paragraph 53 does not correspond to a spatial distance 
separating adjacent objects in the current object cluster. Furthermore, the location or other 
feature that is associated with an image is not "a candidate object interval separating the 
candidate object from an adjacent object in the sequence already segmented into the current 
object cluster," which is compared to the measure of cluster extent in accordance with claim 1. 

For these additional reasons, the Examiner's rejection of claim 5 under 35 U.S.C. § 
102(e) over Piatt should be withdrawn. 

d. Claim 8 

Claim 8 recites that "the processing comprises determining the weighted measures of 
cluster extent by applying to the measures of cluster extent respective weights that decrease with 
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increasing sizes of the respective object clusters." The Examiner has stated that Piatt discloses 
this feature in paragraph 72. Paragraph 72, however, does not support the Examiner's statement. 

In paragraph 72, Piatt discloses that the sorted images then are clustered based on the 
pairwise difference between color histograms of adjacent pairs of the images in accordance with 
Omohundro's best-first model merging technique (see ^ 72; FIG. 10, steps S1005, S1006, and 
S1010-S1013). Contrary to the Examiner's statement, this disclosure does not teach "applying to 
the measures of cluster extent respective weights that decrease with increasing sizes of the 
respective object clusters." Indeed, the merging technique disclosed in paragraph 72 does not 
involve applying weights to measures of cluster extent. 

For this additional reason, the Examiner's rejection of claim 8 under 35 U.S. C. § 102(e) 
over Piatt should be withdrawn. 

e. Claim 10 

Claim 10 recites "customizing at least one of the weights applied to the measures of 
cluster extent based on an analysis of objects in the corresponding object cluster." The Examiner 
has stated that Piatt discloses this feature in paragraph 40. Paragraph 40, however, does not 
support the Examiner's statement. 

In paragraph 40, Piatt describes the "data collection" phase of his cataloging method. 
This phase involves gathering ancillary data before, during or after the capture of a media object. 
The ancillary data may include information added by the user, information from a GPS device, 
or audio added from memory or an external device. This paragraph, however, does not teach 
"customizing at least one of the weights applied to the measures of cluster extent based on an 
analysis of objects in the corresponding object cluster." Indeed, this disclosure has nothing 
whatsoever to do with applying weights to measures of cluster extent. 

For this additional reason, the Examiner's rejection of claim 10 under 35 U.S. C. § 102(e) 
over Piatt should be withdrawn. 

f. Claim 11 

Claim 11 depends from claim 10 and recites that "the customizing comprises scaling at 
least one of the weights applied to the measures of cluster extent based on a fractal dimension 



Applicant : Ullas Gargi 
Serial No. : 10/631,369 
Filed : July 3 1,2003 



Attorney's Docket No.: 200207387-1 
Appeal Brief dated Jan. 17, 2007 
Reply to final action dated Aug. 23, 2006 



Page : 37 of 71 

estimate of recorded time generation meta data associated with the objects in the collection." 
The Examiner has stated that Piatt discloses this feature in paragraphs 40-42. None of the cited 
paragraphs, however, supports the Examiner's statement. 

In paragraph 40, Piatt describes the "data collection" phase of his cataloging method. 
This phase involves gathering ancillary data before, during or after the capture of a media object. 
The ancillary data may include information added by the user, information from a GPS device, 
or audio added from memory or an external device. This paragraph, however, does not teach 
that "the customizing comprises scaling at least one of the weights applied to the measures of 
cluster extent based on a fractal dimension estimate of recorded time generation meta data 
associated with the objects in the collection." Indeed, this disclosure does not teach anything 
whatsoever about a fractal dimension estimate of recorded time generation meta data. 

In paragraph 41, Piatt teaches that during the data collection phase of the cataloging 
method data, such as the date and time of a photograph being taken and sampled audio, may be 
gathered. This paragraph, however, does not teach that "the customizing comprises scaling at 
least one of the weights applied to the measures of cluster extent based on a fractal dimension 
estimate of recorded time generation meta data associated with the objects in the collection," 
Indeed, this disclosure does not teach anything whatsoever about a fractal dimension estimate of 
recorded time generation meta data. 

In paragraph 42, Piatt teaches that during the data correlation phase of the cataloging 
method "one or more pieces of data from the data collection phase are integrated and synthesized 
to form new data that may be used later to identify, organize, and search media objects." This 
paragraph, however, does not teach that "the customizing comprises scaling at least one of the 
weights applied to the measures of cluster extent based on a fractal dimension estimate of 
recorded time generation meta data associated with the objects in the collection." Indeed, this 
disclosure does not teach anything whatsoever about a fractal dimension estimate of recorded 
time generation meta data. 

For these additional reasons, the Examiner's rejection of claim 1 1 under 35 U.S.C. § 
102(e) over Piatt should be withdrawn. 
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g. Claim 12 

Claim 12 recites "customizing at least one of the weights applied to the measures of 
cluster object density based on an analysis of objects in the corresponding object cluster." The 
Examiner has stated that Piatt discloses this feature in paragraph 40. Paragraph 40, however, 
does not support the Examiner's statement. 

In paragraph 40, Piatt describes the "data collection" phase of his cataloging method. 
This phase involves gathering ancillary data before, during or after the capture of a media object. 
The ancillary data may include information added by the user, information from a GPS device, 
or audio added from memory or an external device. This paragraph, however, does not teach 
"customizing at least one of the weights applied to the measures of cluster object density based 
on an analysis of objects in the corresponding object cluster." Indeed, this disclosure has nothing 
whatsoever to do with applying weights to measures of cluster extent. 

For this additional reason, the Examiner's rejection of claim 12 under 35 U.S.C. § 102(e) 
over Piatt should be withdrawn. 

h. Claim 13 

Claim 13 depends from claim 12 and recites that "the customizing comprises scaling at 
least one of the weights applied to the measures of cluster extent based on a fractal dimension 
estimate of recorded time generation meta data associated with the objects in the collection." 
The Examiner has stated that Piatt discloses this feature in paragraph 40. Paragraph 40, 
however, does not support the Examiner's statement. 

In paragraph 40, Piatt describes the "data collection" phase of his cataloging method. 
This phase involves gathering ancillary data before, during or after the capture of a media object. 
The ancillary data may include information added by the user, information from a GPS device, 
or audio added from memory or an external device. This paragraph, however, does not teach 
that "the customizing comprises scaling at least one of the weights applied to the measures of 
cluster extent based on a fractal dimension estimate of recorded time generation meta data 
associated with the objects in the collection." Indeed, this disclosure does not teach anything 
whatsoever about a fractal dimension estimate of recorded time generation meta data. 
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For this additional reason, the Examiner's rejection of claim 13 under 35 U.S.C. § 102(e) 
over Piatt should be withdrawn. 

i. Claim 14 

Claim 14 recites that "the processing further comprises comparing the object density of a 
candidate object cluster consisting of the current object cluster and the candidate object with the 
weighted measure of object density for the current object cluster." The Examiner has stated that 
Piatt discloses this feature in paragraphs 59 and 60, None of the cited paragraphs, however, 
supports the Examiner's statement. 

In paragraph 59, Piatt describes the first (time-based) clustering method. In accordance 
with this method, images are sorted and processed in order of creation time. The creation time of 
each image is compared to a lasttime variable (see K 59; FIG. 9, step S904). If the difference 
between the creation time of the current image and the lasttime variable is less than X hours, the 
image is added to the current collection and the lasttime variable is set equal to the creation time 
of the current image (see fflf 59 and 60; FIG. 9, steps S905 and S906); otherwise, the current 
image is added to a new empty current collection (see | 59; FIG. 9, step S910). The variable X 
may be set to a default value or a value set by the user (see f 59). Alternatively, the value of X 
may be set adaptively in accordance with equation (1), which computes a running average of the 
log time gaps between adjacent ones of the sorted photographs (see equation 1 and K 63). In this 
process, the difference between the creation time of the current image and the lasttime variable 
does not correspond to either (i) the object density of a candidate object cluster consisting of the 
current object cluster and the candidate object or (ii) the weighted measure of object density for 
the current object cluster. Therefore, the comparison that is performed in the method described 
in paragraph 59 does not involve "comparing the object density of a candidate object cluster 
consisting of the current object cluster and the candidate object with the weighted measure of 
object density for the current object cluster," as recited in claim 14. 

In paragraph 60, Piatt discloses that the lasttime variable is updated to reflect the creation 
date of the current image stored in a current collection. This disclosure, however, does not 
make-up for the failure of paragraph 59 to teach "comparing the object density of a candidate 
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object cluster consisting of the current object cluster and the candidate object with the weighted 
measure of object density for the current object cluster," as recited in claim 14. 

For these additional reasons, the Examiner's rejection of claim 14 under 35 ILS.C. § 
102(e) over Piatt should be withdrawn. 



Claim 1 6 recites that "the measure of object density for each current object cluster 
corresponds to an average spatial distance separating adjacent objects in the current object 
cluster." The Examiner has stated that Piatt discloses this feature in paragraphs 45 9 46, and 53. 
None of the cited paragraphs, however, supports the Examiner's statement. 

In paragraph 45, Piatt teaches that the cataloging process may include processing the 
media object with respect to time, date, and/or location, comparing the relevant data with 
threshold data, or performing an inexact search. This disclosure, however, does not teach that 
"the measure of object density for each current object cluster corresponds to an average spatial 
distance separating adjacent objects in the current object cluster." 

In paragraph 46, Piatt describes a process of using a global calendar to catalog a media 
object. In this process, date information for an image to be cataloged is compared to dates in the 
global calendar. If the date corresponds to January 1st, it is inferred that the image was taken on 
New Year's Day and this inference may be used to catalog the image. This disclosure, however, 
does not teach that "the measure of object density for each current object cluster corresponds to 
an average spatial distance separating adjacent objects in the current object cluster." 

In paragraph 53, Piatt describes steps S713-S720 of the method shown in FIG. 7. In 
accordance with this method, if the location associated with an image is within 50 miles of the 
vacation location, then all of the images at the same location are added to the pruned list (see | 
53, FIG. 7, steps 713-S716 and S706). Otherwise, other cataloging processes are performed 
(e.g., add to the pruned list images having features in common with the selected image; see FIG. 
7, steps S717 and S718). Contrary to the Examiner's statement, this disclosure does not teach 
that "the measure of object density for each current object cluster corresponds to an average 
spatial distance separating adjacent objects in the current object cluster." Indeed, the 50-mile 
range used in the example described in paragraph 53 does not correspond to a spatial distance 



Claim 16 



Applicant : Ullas Gargi 
Serial No. : 10/631,369 
Filed : July 31, 2003 



Attorney's Docket No.: 200207387-1 
Appeal Brief dated Jan. 17, 2007 
Reply to final action dated Aug. 23, 2006 



Page : 41 of 71 

separating adjacent objects in the current object cluster. Furthermore, the location or other 
feature that is associated with an image is not "a candidate object interval separating the 
candidate object from an adjacent object in the sequence already segmented into the current 
object cluster/' which is compared to the measure of object density in accordance with claim 1. 

For these additional reasons, the Examiner's rejection of claim 16 under 35 U.S.C. § 
102(e) over Piatt should be withdrawn. 

k. Claim 18 

Claim 1 8 recites that "the processing comprises determining the weighted measures of 
cluster extent by applying to the measures of cluster extent respective weights that decrease with 
increasing sizes of the respective object clusters." The Examiner has stated that Piatt discloses 
this feature in paragraph 72. Paragraph 72, however, does not support the Examiner's statement. 

In paragraph 72, Piatt discloses that the sorted images then are clustered based on the 
pairwise difference between color histograms of adjacent pairs of the images in accordance with 
Omohundro's best-first model merging technique (see ^ 72; FIG. 10, steps S1005, S1006, and 
S1010-S1013). Contrary to the Examiner's statement, this disclosure does not teach "applying to 
the measures of cluster extent respective weights that decrease with increasing sizes of the 
respective object clusters/' Indeed, the merging technique disclosed in paragraph 72 does not 
involve applying weights to measures of cluster extent. 

For this additional reason, the Examiner's rejection of claim 18 under 35 U.S.C. § 102(e) 
over Piatt should be withdrawn. 

5. Independent claim 22 

Independent claim 22 recites features that essentially track the pertinent features of 
independent claim 1 discussed above and therefore is patentable over Piatt for at least the same 
reasons explained above. 

6. Independent claim 23 
Independent claim 23 recites: 



23. A method of organizing a collection of objects, comprising: 
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segmenting objects from the collection into clusters; 

extracting context-related meta data associated with the 
objects and parsable into multiple levels of a name hierarchy; and 

assigning names to clusters based on the extracted context- 
related meta data corresponding to a level of the name hierarchy 
selected to distinguish segmented clusters from one another. 



The Examiner has explained the basis of the rejection of claim 23 over Piatt as follows 
(see page 10, third full paragraph of the final Office action): 



For claim 23, Piatt teaches "a method of organizing a collection of 
objects, comprising segmenting objects from the collection into 
clusters; extracting context-related meta data associated with the 
objects and parsable into multiple levels of a name hierarchy" (See 
paragraph [0037, 0059]); "and assigning names to clusters based 
on the extracted context-related meta data corresponding to a level 
of the name hierarchy selected to distinguish segmented clusters 
from one another" (See paragraph [0035, 0036, 0040, 0046]). 



Contrary to the Examiner's stated rationale, however, none of the cited paragraphs (i.e., 
ffif 35, 36, 40, and 46) teaches "assigning names to clusters based on the extracted context-related 
meta data corresponding to a level of the name hierarchy selected to distinguish segmented 
clusters from one another" 

In paragraph 35, Piatt defines the process of "categorization" in which a user predefines 
labels representing collection, and defines the process of "clustering" in which all objects are 
placed into collection without using predefined labels. This paragraph, however, does not teach 
"assigning names to clusters based on the extracted context-related meta data corresponding to a 
level of the name hierarchy selected to distinguish segmented clusters from one another," as 
recited in claim 23. 

In paragraph 36, Piatt teaches that media objects in a "collection" have similar content or 
relate to similar subject matter and that "Keywords may be used to indicate what information is 
found in a media object stored under a particular collection and/or category or to index a media 
object." This paragraph, however, does not teach "assigning names to clusters based on the 
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extracted context-related meta data corresponding to a level of the name hierarchy selected to 
distinguish segmented clusters from one another," as recited in claim 23. 

In paragraph 40, Piatt describes the "data collection" phase of his cataloging method. 
This phase involves gathering ancillary data before, during or after the capture of a media object. 
The ancillary data may include information added by the user, information from a GPS device, 
or audio added from memory or an external device. This paragraph, however, does not teach 
"assigning names to clusters based on the extracted context-related meta data corresponding to a 
level of the name hierarchy selected to distinguish segmented clusters from one another," as 
recited in claim 23. 

In paragraph 46, Piatt describes a process of using a global calendar to catalog a media 
object. In this process, date information for an image to be cataloged is compared to dates in the 
global calendar. If the date corresponds to January 1 st , it is inferred that the image was taken on 
New Year's Day and this inference may be used to catalog the image. This paragraph, however, 
does not teach "assigning names to clusters based on the extracted context-related meta data 
corresponding to a level of the name hierarchy selected to distinguish segmented clusters from 
one another," as recited in claim 23. Indeed, the cataloging method disclosed in paragraphs 37- 
56, does not form clusters. Instead, the cataloging method only forms an index of the media 
objects in the collection (see, e.g., If 34: "Cataloging refers to forming an index of a collection of 
media objects"; see also 37-56). Note that the pruned list that is created during the correlation 
process is used only to infer information for cataloging the currently selected image (see steps 
S204 and S205 of FIG. 2); the pruned list is not treated as a cluster and is not assigned a name. 
Since clusters are not formed during Piatt's cataloging method, one skilled in the art would not 
have had any reasonable basis for believing that this cataloging method involves "assigning 
names to clusters based on the extracted context-related meta data corresponding to a level of the 
name hierarchy selected to distinguish segmented clusters from one another." 

Therefore, none of the sections of Piatt's disclosure that were cited by the Examiner 
teaches assigning names to clusters based on extracted context-related metadata, as recited in 
claim 23. For at least this reason, the Examiner's rejection of claim 23 under 35 U.S.C. § 102(e) 
over Piatt should be withdrawn. 
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7. Dependent claims 24-3 1 

Each of claims 24-31 incorporates the features of independent claim 23 and therefore is 
patentable over Piatt for at least the same reasons. Claim 29 also is patentable over Platter for 
the following additional reasons. 

Claim 29 recites "automatically naming objects in a given cluster based on the name 
assigned to the given cluster." The Examiner has stated that Piatt discloses this feature in 
paragraphs 35-38. None of the cited paragraphs, however, supports the Examiner's statement. 

In paragraph 35, Piatt defines the process of "categorization" in which a user predefines 
labels representing collection, and defines the process of "clustering" in which all objects are 
placed into collection without using predefined labels. This paragraph, however, does not teach 
"automatically naming objects in a given cluster based on the name assigned to the given 
cluster," as recited in claim 29. Indeed, paragraph 35 does not teach anything about naming 
objects in a given cluster. 

In paragraph 36, Piatt teaches that media objects in a "collection" have similar content or 
relate to similar subject matter and that "Keywords may be used to indicate what information is 
found in a media object stored under a particular collection and/or category or to index a media 
object." This paragraph, however, does not teach "automatically naming objects in a given 
cluster based on the name assigned to the given cluster," as recited in claim 29. Indeed, 
paragraph 36 does not teach anything about naming objects in a given cluster. 

In paragraph 37, Piatt teaches that media objects may be cataloged based upon correlation 
of the data for the media object and/or data describing the media object (metadata, such as data, 
time, location) with known data and metadata. This disclosure does not teach "automatically 
naming objects in a given cluster based on the name assigned to the given cluster," as recited in 
claim 29. Indeed, paragraph 37 does not teach anything about naming objects in a given cluster. 

In paragraph 38, Piatt teaches that the cataloging method involves "taking known data 
that has been associated with attributes and labeled, for example, with 'Grandmother's house' 
and using that known data to infer a label for other data." This disclosure does not teach 
"automatically naming objects in a given cluster based on the name assigned to the given 
cluster," as recited in claim 29. Indeed, paragraph 38 does not teach anything about naming 
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objects in a given cluster. The process of associating labels with objects for indexing purposes 
does not constitute naming objects (see, e.g., 34, lines 6-8: "Indexing refers to building a table 
or index that includes pointers to data"). 

For these additional reasons, the Examiner's rejection of claim 29 under 35 U.S.C. § 
102(e) over Piatt should be withdrawn. 

8. Independent claim 32 

Independent claim 32 recites features that essentially track the pertinent features of 
independent claim 23 discussed above and therefore is patentable over Piatt for at least the same 
reasons explained above. 

9, Independent claim 33 

Independent claim 33 recites: 

33. A method of organizing a collection of objects, comprising: 

accessing a sequence of objects segmented into clusters 
each including multiple constituent objects arranged in a respective 
sequence in accordance with context-related meta data associated 
with the objects; 

selecting for each object cluster at least two constituent 
objects representative of beginning and ending instances in the 
corresponding object sequence; and 

in a user interface, graphically presenting the selected 
representative objects of each cluster without graphically 
presenting representations of unselected ones of the constituent 
objects of the clusters. 

The Examiner has explained the basis of the rejection of claim 33 over Piatt as follows 
(see page 13, first full paragraph of the final Office action): 



For claim 33, Piatt teaches "accessing a sequence of objects 
segmented into clusters each including multiple objects arranged in 
a respective sequence in accordance with context-related meta data 
associated with the objects"(See paragraph [0034-0037]); 
"selecting for each object cluster at least two constituent objects 
representative of beginning and ending instances in the 
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corresponding object sequence"(See paragraph [0059-0060]); "and 
in a user interface graphically presenting the selected 
representative objects of each cluster without graphically 
presenting representations of unselected ones of the constituent 
object of the clusters" (See paragraph [0072-0077]). 

Contrary to the Examiner's stated rationale, however, neither paragraph 59 nor paragraph 
60 teaches "selecting for each object cluster at least two constituent objects representative of 
beginning and ending instances in the corresponding object sequence." In paragraph 59, Piatt 
describes the process of automatically organizing media objects according to their creation times 
in accordance with the first (time-based) clustering method. In this paragraph, Piatt discloses 
that either the first (earliest created) or latest (last created) image in some predetermined 
directory structure is selected to begin the first clustering method. This paragraph, however, 
does not teach anything about selecting for each object cluster at least two constituent objects 
representative of beginning and ending instances in the corresponding object sequence. In 
paragraph 60, Piatt describes selecting a single representative image for a current collection that 
results from the first (time-based) clustering method. In the only example disclosed in paragraph 
60, the representative image is the image with the median creation time. 

In addition, none of the cited paragraphs 72-77 teaches "in a user interface graphically 
presenting the selected representative objects of each cluster without graphically presenting 
representations of unselected ones of the constituent object of the clusters." 

Paragraphs 72-74 describe steps of the second (color-based) cluster algorithm shown in 
FIG. 10. In paragraph 72, Piatt discloses that the sorted images then are clustered based on the 
pairwise difference between color histograms of adjacent pairs of the images in accordance with 
Omohundro's best-first model merging technique (see K 72; FIG. 10, steps S1005, S1006, and 
S1010-S1013). In paragraph 73, Piatt discloses that a single representative image is chosen for 
each cluster that results from the second (color-based) clustering method. In paragraph 74, Piatt 
discloses that "The representative image is the image whose color best mimics the overall color 
of the cluster." 

Paragraph 75 describes a method that combines temporal and color clustering of 
photographs (see ^ 75, lines 2-3). In step SI 108 of this method, a single representative image is 
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computed for each cluster using a method analogous to the method described in connection with 
step SI 107 of the second (color-based) clustering method (see f 75, lines 27-29). 

Paragraph 76 describes the graphical user interface shown in FIG. 12. In this user 
interface, a single representative image is shown for each cluster in the left pane, and all the 
images in a cluster corresponding to a selected one of the representative images are shown in the 
right pane. 

Paragraph 77 describes the graphical user interface shown in FIG. 13. In this user 
interface, the right pane shows thumbnails 1303 of all the user's photographs and the left pane 
shows a single representative image for each cluster. When the user selects one of the 
representative images in the left pane, the scrollbar 1304 is scrolled so that the same thumbnail is 
vertically centered in the right pane and highlighted with a colored border (see 1 77, lines 10-15). 

Thus, none of the cited paragraphs 72-77 teaches "in a user interface graphically 
presenting the selected representative objects of each cluster without graphically presenting 
representations of unselected ones of the constituent object of the clusters," as recited in claim 
33. In fact, Piatt does not teach such a step anywhere in his entire disclosure. 

For at least these reasons, the Examiner's rejection of claim 33 under 35 U.S.C. § 102(e) 
over Piatt should be withdrawn. 

10. Dependent claims 34-50 

Each of claims 34-50 incorporates the features of independent claim 33 and therefore is 
patentable over Piatt for at least the same reasons. Claims 34, 36-39, 41-43, and 50 also are 
patentable over Piatt for the following additional reasons. 

a. Claim 34 

Claim 34 recites "graphically presenting a selected one of the clusters as a stack of 
partially overlapping images representative of multiple objects in the selected cluster." The 
Examiner has stated that Piatt discloses this feature in paragraphs 76 and 77. None of the cited 
paragraphs, however, supports the Examiner's statement. 

In paragraph 76, Piatt describes the graphical user interface shown in FIG. 12. In this 
user interface, a single representative image is shown for each cluster in the left pane, and all the 
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images in a cluster corresponding to a selected one of the representative images are shown in the 
right pane. This disclosure does not teach "graphically presenting a selected one of the clusters 
as a stack of partially overlapping images representative of multiple objects in the selected 
cluster," as recited in claim 34. 

In Paragraph 77, Piatt describes the graphical user interface shown in FIG. 13. In this 
user interface, the right pane shows thumbnails 1303 of all the user's photographs and the left 
pane shows a single representative image for each cluster. When the user selects one of the 
representative images in the left pane, the scrollbar 1304 is scrolled so that the same thumbnail is 
vertically centered in the right pane and highlighted with a colored border (see K 77, lines 10-15). 
This disclosure does not teach "graphically presenting a selected one of the clusters as a stack of 
partially overlapping images representative of multiple objects in the selected cluster," as recited 
in claim 34. 

For these additional reasons, the Examiner's rejection of claim 34 under 35 U.S.C. § 
102(e) over Piatt should be withdrawn. 

b. Claim 36 

Claim 36 recites that "the presenting comprises presenting the selected representative 
objects with the spacing between adjacent ones of the selected representative objects in the same 
cluster smaller than the spacing between adjacent ones of the selected representative objects in 
different clusters." The Examiner has stated that Piatt discloses this feature in paragraphs 72-77. 
None of the cited paragraphs, however, supports the Examiner's statement. 

Paragraphs 72-74 describe steps of the second (color-based) cluster algorithm shown in 
FIG. 10. 

In paragraph 72, Piatt discloses that the sorted images then are clustered based on the 
pairwise difference between color histograms of adjacent pairs of the images in accordance with 
Omohundro's best-first model merging technique (see If 72; FIG. 10, steps SI 005, SI 006, and 
S1010-S1013). This disclosure does not teach anything about how to present representative 
objects. 

In paragraph 73, Piatt discloses that a single representative image is chosen for each 
cluster that results from the second (color-based) clustering method. In paragraph 74, Piatt 



Applicant : Ullas Gargi 
Serial No. : 10/631,369 
Filed : July 31, 2003 



Attorney's Docket No,: 200207387-1 
Appeal Brief dated Jan. 17, 2007 
Reply to final action dated Aug. 23, 2006 



Page : 49 of 71 

discloses that "The representative image is the image whose color best mimics the overall color 
of the cluster." This disclosure does not teach anything about how to present representative 
objects. 

Paragraph 75 describes a method that combines temporal and color clustering of 
photographs (see ^ 75, lines 2-3). In step SI 108 of this method, a single representative image is 
computed for each cluster using a method analogous to the method described in connection with 
step SI 107 of the second (color-based) clustering method (see ^ 75, lines 27-29). This disclosure 
does not teach anything about how to present representative objects. 

In paragraph 76, Piatt describes the graphical user interface shown in FIG. 12. In this 
user interface, a single representative image is shown for each cluster in the left pane, and all the 
images in a cluster corresponding to a selected one of the representative images are shown in the 
right pane. This disclosure does not teach presenting the selected representative objects with the 
spacing between adjacent ones of the selected representative objects in the same cluster smaller 
than the spacing between adjacent ones of the selected representative objects in different 
clusters," as recited in claim 36. 

In Paragraph 77, Piatt describes the graphical user interface shown in FIG. 13. In this 
user interface, the right pane shows thumbnails 1303 of all the user's photographs and the left 
pane shows a single representative image for each cluster. When the user selects one of the 
representative images in the left pane, the scrollbar 1304 is scrolled so that the same thumbnail is 
vertically centered in the right pane and highlighted with a colored border (see ^ 77, lines 10-15). 
This disclosure does not teach presenting the selected representative objects with the spacing 
between adjacent ones of the selected representative objects in the same cluster smaller than the 
spacing between adjacent ones of the selected representative objects in different clusters," as 
recited in claim 36. 

For these additional reasons, the Examiner's rejection of claim 36 under 35 U.S.C. § 
102(e) over Piatt should be withdrawn. 
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c. Claim 37 

Claim 37 recites "merging objects of one cluster into an adjacent cluster in response to 
user input." The Examiner has stated that Piatt discloses this feature in paragraph 76. Paragraph 
76, however, does not support the Examiner's statement. 

In paragraph 76, Piatt describes the graphical user interface shown in FIG. 12. In this 
user interface, a single representative image is shown for each cluster in the left pane, and all the 
images in a cluster corresponding to a selected one of the representative images are shown in the 
right pane. The user can scroll through the representative images. When the user clicks on a 
representative image in the left pane, the contents of that cluster are shown in the right pane. 
When the user clicks on a thumbnail in the right pane, the user is shown a full-sized version of 
the image. Paragraph 76 also discloses that "if the user drags or copies a thumbnail from 1203, 
the full-sized image can be a drop candidate or can be put on a clipboard, respectively. 
Paragraph 76, however, does not teach "merging objects of one cluster into an adjacent cluster in 
response to user input," as recited in claim 37. 

For this additional reason, the Examiner's rejection of claim 37 under 35 U.S.C. § 102(e) 
over Piatt should be withdrawn. 

d. Claim 38 

Claim 38 depends from claim 37 and recites that "objects of one cluster are merged into 
an adjacent cluster in response to dragging and dropping of the objects to be merged." The 
Examiner has stated that Piatt discloses this feature in paragraph 76. Paragraph 76, however, 
does not support the Examiner's statement. 

In paragraph 76, Piatt describes the graphical user interface shown in FIG. 12. In this 
user interface, a single representative image is shown for each cluster in the left pane, and all the 
images in a cluster corresponding to a selected one of the representative images are shown in the 
right pane. The user can scroll through the representative images. When the user clicks on a 
representative image in the left pane, the contents of that cluster are shown in the right pane. 
When the user clicks on a thumbnail in the right pane, the user is shown a full-sized version of 
the image. Paragraph 76 also discloses that "if the user drags or copies a thumbnail from 1203, 
the full-sized image can be a drop candidate or can be put on a clipboard, respectively. 
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Paragraph 76, however, does not teach that "objects of one cluster are merged into an adjacent 
cluster in response to dragging and dropping of the objects to be merged," as recited in claim 38. 

For this additional reason, the Examiner's rejection of claim 38 under 35 U.S.C. § 102(e) 
over Piatt should be withdrawn. 

e. Claim 39 

Claim 39 depends from claim 37 and recites that "the objects of the one cluster are 
merged into the adjacent cluster in response to user selection of an icon for merging the 
clusters." The Examiner has stated that Piatt discloses this feature in paragraph 76. Paragraph 
76, however, does not support the Examiner's statement. 

In paragraph 76, Piatt describes the graphical user interface shown in FIG. 12. In this 
user interface, a single representative image is shown for each cluster in the left pane, and all the 
images in a cluster corresponding to a selected one of the representative images are shown in the 
right pane. The user can scroll through the representative images. When the user clicks on a 
representative image in the left pane, the contents of that cluster are shown in the right pane. 
When the user clicks on a thumbnail in the right pane, the user is shown a full-sized version of 
the image. Paragraph 76 also discloses that "if the user drags or copies a thumbnail from 1203, 
the full-sized image can be a drop candidate or can be put on a clipboard, respectively. 
Paragraph 76, however, does not teach that "the objects of the one cluster are merged into the 
adjacent cluster in response to user selection of an icon for merging the clusters," as recited in 
claim 39. 

For this additional reason, the Examiner's rejection of claim 39 under 35 U.S.C. § 102(e) 
over Piatt should be withdrawn. 

f Claim 41 

Claim 41 depends from claim 40 and recites that an "object distribution for a given 
cluster is presented as object instances plotted along an axis corresponding to a scaled 
representation of the context-related extent spanned by the given cluster." The Examiner has 
stated that Piatt discloses this feature in paragraphs 64-73. None of the cited paragraphs, 
however, supports the Examiner's statement. 
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In paragraphs 64-72, Piatt discloses aspects of the second (color-based) clustering method 
shown in FIG. 10 and described above in § VILB.L 

In paragraph 64, Piatt describes step SI 001 in which color histograms of all of the images 
are created. Contrary to the Examiner's statement, Piatt's disclosure in paragraph 64 does not 
teach that an "object distribution for a given cluster is presented as object instances plotted along 
an axis corresponding to a scaled representation of the context-related extent spanned by the 
given cluster," as recited in claim 41. 

In paragraph 65, Piatt describes how the color histograms may be estimated. Contrary to 
the Examiner's statement, Piatt's disclosure in paragraph 65 does not teach that an "object 
distribution for a given cluster is presented as object instances plotted along an axis 
corresponding to a scaled representation of the context-related extent spanned by the given 
cluster," as recited in claim 41. 

In paragraph 66, Piatt describes step SI 002 in which the creation times of all of the 
images are extracted. Contrary to the Examiner's statement, Piatt's disclosure in paragraph 66 
does not teach that an "object distribution for a given cluster is presented as object instances 
plotted along an axis corresponding to a scaled representation of the context-related extent 
spanned by the given cluster," as recited in claim 41. 

In paragraph 67, Piatt describes step SI 003 in which all the images are sorted by creation 
time. Contrary to the Examiner's statement, Piatt's disclosure in paragraph 67 does not teach 
that an "object distribution for a given cluster is presented as object instances plotted along an 
axis corresponding to a scaled representation of the context-related extent spanned by the given 
cluster," as recited in claim 41. 

In paragraph 68, Piatt describes step SI 004 in which the pairwise distances between the 
color histograms of adjacent pairs of ordered images are computed. Contrary to the Examiner's 
statement, Piatt's disclosure in paragraph 68 does not teach that an "object distribution for a 
given cluster is presented as object instances plotted along an axis corresponding to a scaled 
representation of the context-related extent spanned by the given cluster," as recited in claim 41. 

In paragraph 69, Piatt describes aspects of equations (2) and (3). Contrary to the 
Examiner's statement, Piatt's disclosure in paragraph 69 does not teach that an "object 
distribution for a given cluster is presented as object instances plotted along an axis 
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corresponding to a scaled representation of the context-related extent spanned by the given 
cluster," as recited in claim 41 . 

In paragraph 70, Piatt describes how the distance between two images or clusters can be 
determined. Contrary to the Examiner's statement, Piatt's disclosure in paragraph 70 does not 
teach that an "object distribution for a given cluster is presented as object instances plotted along 
an axis corresponding to a scaled representation of the context-related extent spanned by the 
given cluster," as recited in claim 41. 

In paragraph 71, Piatt discloses that distance metric other than the one defined in 
equation (4) may be used in step SI 004. Contrary to the Examiner's statement, Piatt's disclosure 
in paragraph 71 does not teach that an "object distribution for a given cluster is presented as 
object instances plotted along an axis corresponding to a scaled representation of the context- 
related extent spanned by the given cluster," as recited in claim 41. 

In paragraph 72, Piatt discloses that the sorted images then are clustered based on the 
pairwise difference between color histograms of adjacent pairs of the images in accordance with 
Omohundro's best-first model merging technique (see ^ 72; FIG. 10, steps SI 005, SI 006, and 
S1010-S1013). According to Piatt (t 72): 

... The technique starts with each item in its own cluster. Then, at 
every step, the distance between every pair of adjacent clusters is 
computed. The pair with the smallest distance is merged together 
to yield a new cluster. This new cluster replaces the two old 
clusters in the sequence, and the clustering continues until the 
desired number of clusters is reached. ... 

Contrary to the Examiner's statement, Piatt's disclosure in paragraph 72 does not teach that an 
"object distribution for a given cluster is presented as object instances plotted along an axis 
corresponding to a scaled representation of the context-related extent spanned by the given 
cluster," as recited in claim 41 . 

In paragraph 73, Piatt discloses that a single representative image is chosen for each 
cluster that results from the second (color-based) clustering method. Contrary to the Examiner's 
statement, Piatt's disclosure in paragraph 73 does not teach that an "object distribution for a 
given cluster is presented as object instances plotted along an axis corresponding to a scaled 
representation of the context-related extent spanned by the given cluster," as recited in claim 41. 
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For these additional reasons, the Examiner's rejection of claim 41 under 35 U.S.C. § 
102(e) over Piatt should be withdrawn. 

g. Claim 42 

Claim 42 recites "splitting a given cluster in response to user selection of a point in the 
representation of the object distribution presented for the given cluster." The Examiner has 
stated that Piatt discloses this feature in paragraph 75. Paragraph 75, however, does not support 
the Examiner's statement. 

In paragraph 75, Piatt describes a method that combines temporal and color clustering of 
photographs (see f 75, lines 2-3). In this method, the images first are clustered in accordance 
with the first (time-based) clustering method shown in FIG. 9 (see | 75; FIG. 1 1, step SI 102). 
The resulting clusters "are scanned in steps SI 103-S1 107 to check for failure of the temporal 
clustering" (see | 75, lines 9-1 1). Overly large clusters are identified by comparing the number 
of images in the clusters to a threshold (see f 75, lines 11-12). If the number is greater than the 
threshold for a current cluster, the current cluster is split in to sub-clusters in accordance with the 
second (color-based) clustering algorithm shown in FIG. 10 (see 1 75, lines 12-25). Thus, in 
accordance with Piatt's teachings, the current cluster is split in response to a determination that 
the number of images in the cluster exceeds a threshold. Therefore, paragraph 75 does not teach 
"splitting a given cluster in response to user selection of a point in the representation of the 
object distribution presented for the given cluster," as recited in claim 42. 

For this additional reason, the Examiner's rejection of claim 42 under 35 U.S.C. § 102(e) 
over Piatt should be withdrawn. 

h. Claim 43 

Claim 43 recites "automatically splitting a given cluster into two or more clusters in 
response to user input." The Examiner has stated that Piatt discloses this feature in paragraph 75. 
Paragraph 75, however, does not support the Examiner's statement. 

In paragraph 75, Piatt describes a method that combines temporal and color clustering of 
photographs (see | 75, lines 2-3). In this method, the images first are clustered in accordance 
with the first (time-based) clustering method shown in FIG. 9 (see 1 75; FIG. 1 1, step SI 102). 



Attorney's Docket No.: 200207387-1 
Appeal Brief dated Jan. 17, 2007 
Reply to final action dated Aug. 23, 2006 



Applicant : UllasGargi 
Serial No. : 10/631,369 
Filed : July 3 1,2003 



Attorney's Docket No.: 200207387-1 
Appeal Brief dated Jan. 17, 2007 
Reply to final action dated Aug. 23, 2006 



Page : 55 of 71 

The resulting clusters "are scanned in steps SI 103-S1 107 to check for failure of the temporal 
clustering" (see K 75, lines 94 1). Overly large clusters are identified by comparing the number 
of images in the clusters to a threshold (see 1 75, lines 11-12). If the number is greater than the 
threshold for a current cluster, the current cluster is split in to sub-clusters in accordance with the 
second (color-based) clustering algorithm shown in FIG. 10 (see U 75, lines 12-25), Thus, in 
accordance with Piatt's teachings, the current cluster is split in response to a determination that 
the number of images in the cluster exceeds a threshold. Therefore, paragraph 75 does not teach 
"automatically splitting a given cluster into two or more clusters in response to user input," as 
recited in claim 43. 

For this additional reason, the Examiner's rejection of claim 43 under 35 U.S.C. § 102(e) 
over Piatt should be withdrawn. 

i. Claim 50 

Claim 50 recites "graphically presenting at least one link to an object of a cluster 
arranged in a sequence in accordance with location-related meta data in a map format." The 
Examiner has stated that Piatt discloses this feature in paragraphs 40, 76, and 77. None of the 
cited paragraphs, however, supports the Examiner's statement. 

In paragraph 40, Piatt describes the "data collection" phase of his cataloging method. 
This phase involves gathering ancillary data before, during or after the capture of a media object. 
The ancillary data may include information added by the user, information from a GPS device, 
or audio added from memory or an external device. This paragraph, however, does not teach 
"graphically presenting at least one link to an object of a cluster arranged in a sequence in 
accordance with location-related meta data in a map format," as recited in claim 50. 

In paragraph 76, Piatt describes the graphical user interface shown in FIG. 12. In this 
user interface, a single representative image is shown for each cluster in the left pane, and 
thumbnails of all the images in a cluster corresponding to a selected one of the representative 
images are shown in the right pane. The thumbnails are sorted by increasing creation time (see ^ 
76, lines 6-7). This disclosure does not teach "graphically presenting at least one link to an 
object of a cluster arranged in a sequence in accordance with location-related meta data in a map 
format," as recited in claim 50. 
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In Paragraph 77, Piatt describes the graphical user interface shown in FIG. 13. In this 
user interface, the right pane shows thumbnails 1303 of all the user's photographs and the left 
pane shows a single representative image for each cluster. The thumbnails are sorted by 
increasing creation time (see \ 77, lines 4-5). When the user selects one of the representative 
images in the left pane, the scrollbar 1304 is scrolled so that the same thumbnail is vertically 
centered in the right pane and highlighted with a colored border (see ^ 77, lines 10-15). This 
disclosure does not teach "graphically presenting at least one link to an object of a cluster 
arranged in a sequence in accordance with location-related meta data in a map format," as recited 



For these additional reasons, the Examiner's rejection of claim 50 under 35 U.S.C. § 
102(e) over Piatt should be withdrawn. 

1 1 . Independent claim 5 1 

Independent claim 51 recites features that essentially track the pertinent features of 
independent claim 33 discussed above and therefore is patentable over Piatt for at least the same 
reasons explained above. 

VIIL Conclusion 



in claim 50. 



For the reasons explained above, all of the pending claims are now in condition for 
allowance and should be allowed. 

Charge any excess fees or apply any credits to Deposit Account No. 08-2025. 
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CLAIMS APPENDIX 



The claims that are the subject of Appeal are presented below. 

Claim 1 (previously presented): A method of organizing a collection of objects arranged 
in a sequence ordered in accordance with a selected dimension of context-related metadata 
respectively associated with the objects, comprising: 

classifying the objects in the sequence to generate a series of object clusters, wherein the 
classifying comprises sequentially processing each of the objects as a respective candidate for 
segmentation into a respective current one of the object clusters in the series and, for each of the 
candidate objects, 



determining a candidate object interval separating the candidate object from an 
adjacent object in the sequence already segmented into the current object cluster, 
the candidate object interval being measured in the selected dimension of the 
context-related metadata, 

comparing the candidate object interval to a weighted measure of cluster extent for 
the current object cluster, the measure of cluster extent corresponding to a current 
distance spanned by all the objects in the current object cluster measured in the 
selected dimension of the context-related metadata, and 

comparing the candidate object interval to a weighted measure of object density for 
the current object cluster, the measure of object density corresponding to a 
measure of distribution of distances separating adjacent ones of the objects in the 
current object cluster measured in the selected dimension of the context-related 
metadata. 
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Claim 2 (previously presented): The method of claim 1 , wherein the measure of cluster 
extent for each current object cluster corresponds to a temporal distance spanned by recorded 
generation times associated with all objects in the current object cluster. 

Claim 3 (previously presented): The method of claim 1, wherein the measure of cluster 
extent for each current object cluster corresponds to a spatial distance spanned by recorded 
generation locations associated with all objects in the current object cluster. 

Claim 4 (previously presented): The method of claim 1, wherein the measure of object 
density for each current object cluster corresponds to an average temporal distance separating 
adjacent objects in the current object cluster. 

Claim 5 (previously presented): The method of claim 1, wherein the measure of object 
density for each current object cluster corresponds to an average spatial distance separating 
adjacent objects in the current object cluster. 

Claim 6 (previously presented): The method of claim 1, wherein the classifying 
comprises merging consecutive ones of the candidate objects into a current one of the object 
clusters until the candidate object interval determined for a current one of the candidate objects 
exceeds the weighted measure of cluster extent for the current cluster, at which point a 
successive one of the object clusters in the series is initiated with the current candidate object. 
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Claim 7 (previously presented): The method of claim 1, wherein the classifying 
comprises merging consecutive ones of the candidate objects into a current one of the object 
clusters until the candidate object interval determined for a current one of the candidate objects 
exceeds the weighted measure of object density for the current object cluster, at which point a 
successive one of the object clusters in the series is initiated with the current candidate object. 

Claim 8 (previously presented): The method of claim 1, wherein the processing 
comprises determining the weighted measures of cluster extent by applying to the measures of 
cluster extent respective weights that decrease with increasing sizes of the respective object 
clusters. 

Claim 9 (previously presented): The method of claim 1, wherein the processing 
comprises determining the weighted measures of cluster extent by applying to the measures of 
cluster extent respective weights that decrease with increasing sizes of the respective object 
clusters. 



Claim 10 (previously presented): The method of claim 1, further comprising customizing 
at least one of the weights applied to the measures of cluster extent based on an analysis of 
objects in the corresponding object cluster. 
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Claim 1 1 (previously presented): The method of claim 10, wherein the customizing 
comprises scaling at least one of the weights applied to the measures of cluster extent based on a 
fractal dimension estimate of recorded time generation meta data associated with the objects in 
the collection. 

Claim 12 (previously presented): The method of claim 1, further comprising customizing 
at least one of the weights applied to the measures of cluster object density based on an analysis 
of objects in the corresponding object cluster. 

Claim 13 (previously presented): The method of claim 12, wherein the customizing 
comprises scaling at least one of the weights applied to the measures of cluster extent based on a 
fractal dimension estimate of recorded time generation meta data associated with the objects in 
the collection. 

Claim 14 (previously presented): The method of claim 1, wherein the processing further 
comprises comparing the object density of a candidate object cluster consisting of the current 
object cluster and the candidate object with the weighted measure of object density for the 
current object cluster. 

Claim 15 (previously presented): The method of claim 14, wherein the measure of object 
density for each current object cluster corresponds to an average temporal distance separating 
adjacent objects in the current object cluster. 
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Claim 16 (previously presented): The method of claim 14, wherein the measure of object 
density for each current object cluster corresponds to an average spatial distance separating 
adjacent objects in the current object cluster. 

Claim 17 (previously presented): The method of claim 14, wherein the measure of object 
density for each object cluster corresponds to a moving average distance separating adjacent 
objects in the current object cluster. 

Claim 18 (previously presented): The method of claim 14, wherein the processing 
comprises determining the weighted measures of cluster extent by applying to the measures of 
cluster extent respective weights that decrease with increasing sizes of the respective object 
clusters. 

Claim 1 9 (previously presented): The method of claim 1 , wherein the processing 
comprises processing each of the candidate objects sequentially beginning at a first end of the 
object sequence. 

Claim 20 (previously presented): The method of claim 1 9, wherein the processing further 
comprises processing each of the candidate objects sequentially beginning at a second end of the 
object sequence opposite the first end. 
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Claim 21 (previously presented): The method of claim 1, wherein the sequence to be 
segmented includes objects of the following types: text, audio, graphics, still images, video and 
business events. 

Claim 22 (previously presented): A system of organizing a collection of objects arranged 
in a sequence ordered in accordance with a selected dimension of context-related metadata 
respectively associated with the objects, comprising: 

a segmentation engine operable to classify the objects in the sequence to generate a series 
of object clusters, wherein the segmentation engine is operable to sequentially process each of 
the objects as a respective candidate for segmentation into a respective current one of the object 
clusters in the series and, for each of the candidate objects, perform operations comprising 



determining a candidate object interval separating the candidate object from an 
adjacent object in the sequence already segmented into the current object cluster, 
the candidate object interval being measured in the selected dimension of the 
context-related metadata, 

compare the candidate object interval to a weighted measure of cluster extent for the 
current object cluster, the measure of cluster extent corresponding to a current 
distance spanned by all the objects in the current object cluster measured in the 
selected dimension of the context-related metadata, and 

comparing the candidate object interval to a weighted measure of cluster object 
density for the current object cluster, the measure of object density corresponding 
to a measure of distribution of distances separating adjacent ones of the objects in 
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the current object cluster measured in the selected dimension of the context- 
related metadata. 

Claim 23 (original): A method of organizing a collection of objects, comprising: 
segmenting objects from the collection into clusters; 

extracting context-related meta data associated with the objects and parsable into multiple 
levels of a name hierarchy; and 

assigning names to clusters based on the extracted context-related meta data 
corresponding to a level of the name hierarchy selected to distinguish segmented clusters from 
one another. 

Claim 24 (original): The method of claim 23, wherein names are assigned to clusters 
based on the extracted context-related meta data corresponding to a highest level of the name 
hierarchy that distinguishes clusters from each other. 

Claim 25 (original): The method of claim 23, wherein the context-related meta data 
corresponds to object generation times. 



Claim 26 (original): The method of claim 23, wherein the context-related meta data 
corresponds to object generation locations. 
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Claim 27 (original): The method of claim 26, wherein the context-related meta data 
corresponds to recorded information relating to country, city, and state of object generation. 

Claim 28 (original): The method of claim 23, wherein the context-related meta data 
corresponds to both object generation times and object generation locations. 

Claim 29 (original): The method of claim 23, further comprising automatically naming 
objects in a given cluster based on the name assigned to the given cluster. 

Claim 30 (original): The method of claim 29, wherein the objects in the given cluster are 
named automatically in accordance with a chronological ordering of the objects in the given 
cluster. 

Claim 31 (original): The method of claim 29, further comprising storing objects in the 
given cluster in a tree structure organized by cluster and labeled in accordance with the assigned 
names. 

Claim 32 (original): A system of organizing a collection of objects, comprising: 
a segmentation engine operable to segment objects from the collection into clusters; and 
a naming engine operable to extract context-related meta data associated with the objects 
and parsable into multiple levels of a name hierarchy, and assign names to each cluster based on 
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the extracted context-related meta data corresponding to a level of the name hierarchy selected to 
distinguish segmented clusters from one another. 

Claim 33 (previously presented): A method of organizing a collection of objects, 
comprising: 

accessing a sequence of objects segmented into clusters each including multiple 
constituent objects arranged in a respective sequence in accordance with context-related meta 
data associated with the objects; 

selecting for each object cluster at least two constituent objects representative of 
beginning and ending instances in the corresponding object sequence; and 

in a user interface, graphically presenting the selected representative objects of each 
cluster without graphically presenting representations of unselected ones of the constituent 
objects of the clusters. 

Claim 34 (previously presented): The method of claim 33, further comprising 
graphically presenting a selected one of the clusters as a stack of partially overlapping images 
representative of multiple objects in the selected cluster. 



Claim 35 (previously presented): The method of claim 34, further comprising revealing 
an increased portion of a given one of the representative images in the stack in response to 
detection of a user-controlled display icon positioned over the given representative image. 
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Claim 36 (previously presented): The method of claim 33, wherein the presenting 
comprises presenting the selected representative objects with the spacing between adjacent ones 
of the selected representative objects in the same cluster smaller than the spacing between 
adjacent ones of the selected representative objects in different clusters. 

Claim 37 (original): The method of claim 33, further comprising merging objects of one 
cluster into an adjacent cluster in response to user input. 

Claim 38 (original): The method of claim 37, wherein objects of one cluster are merged 
into an adjacent cluster in response to dragging and dropping of the objects to be merged. 

Claim 39 (original): The method of claim 37, wherein the objects of the one cluster are 
merged into the adjacent cluster in response to user selection of an icon for merging the clusters. 

Claim 40 (original): The method of claim 33, further comprising presenting a graphical 
representation of distributions of objects in the clusters. 



Claim 41 (original): The method of claim 40, wherein a object distribution for a given 
cluster is presented as object instances plotted along an axis corresponding to a scaled 
representation of the context-related extent spanned by the given cluster. 
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Claim 42 (original): The method of claim 40, further comprising splitting a given cluster 
in response to user selection of a point in the representation of the object distribution presented 
for the given cluster. 

Claim 43 (original): The method of claim 40, further comprising automatically splitting a 
given cluster into two or more clusters in response to user input. 

Claim 44 (original): The method of claim 43, wherein the given cluster is automatically 
split into a user-selected number of sub-clusters. 

Claim 45 (original): The method of claim 43, wherein the given cluster is automatically 
split based on relative sizes of intervals between successive objects in the given cluster. 

Claim 46 (original): The method of claim 33, wherein the context-related meta data 
corresponds to object generation times. 

Claim 47 (original): The method of claim 33, wherein the context-related meta data 
corresponds to object generation locations. 



Claim 48 (original: The method of claim 33, wherein the segmented sequence includes 
objects of the following types: text, audio, graphics, still images, video, and business events. 



Applicant : Ullas Gargi 
Serial No. : 10/631,369 
Filed : July 31, 2003 



Attorney's Docket No.: 200207387-1 
Appeal Brief dated Jan. 17, 2007 
Reply to final action dated Aug. 23, 2006 



Page : 69 of 71 

Claim 49 (original): The method of claim 33, further comprising graphically presenting 
at least one link to an object of a cluster arranged in a sequence in accordance with time-related 
meta data in a calendar format 

Claim 50 (original): The method of claim 33, further comprising graphically presenting 
at least one link to an object of a cluster arranged in a sequence in accordance with location- 
related meta data in a map format 

Claim 51 (previously presented): A system of organizing a collection of objects, 
comprising a user interface layout engine operable to perform operations comprising: 

accessing a sequence of objects from the collection segmented into clusters each 
including multiple objects arranged in a respective sequence in accordance with context-related 
meta data associated with the objects; 

selecting for each object cluster at least two constituent objects representative of 
beginning and ending instances in the corresponding object sequence; and 

in a user interface, graphically presenting the selected representative objects of each 
cluster on a screen without graphically presenting representations of unselected ones of the 
constituent objects of the clusters, wherein the user interface layout engine presents the selected 
representative objects with the spacing between adjacent ones of the selected representative 
objects in the same cluster smaller than the spacing between adjacent ones of the selected 
representative objects in different clusters. 
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EVIDENCE APPENDIX 



There is no evidence submitted pursuant to 37 CFR §§ 1.130, 1.131, or 1.132 or any 
other evidence entered by the Examiner and relied upon by Appellant in the pending appeal. 
Therefore, no copies are required under 37 CFR § 41.37(c)(l)(ix) in the pending appeal. 
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RELATED PROCEEDINGS APPENDIX 



Appellant is not aware of any decisions rendered by a court or the Board that will directly 
affect or be directly affected by or have a bearing on the Board's decision in the pending appeal. 
Therefore, no copies are required under 37 CFR § 4L37(c)(l)(x) in the pending appeal. 



